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Preface 



The purpose of this monograph is to provide a # concise 
, 'introduction to the theory of generalized inverses of 
matrices that is accessible to undergraduate mathematics 
majors. Although results from this active area qf research, 
have appeared' in a number of exceJUen£ graduate level text- 
books since 1971, material for use at the undergraduate 
level remains fragmented. The basic ideas are so fundamental, 
however, "that they can be used to unify various topics that* 
b an undergraduate has seen but perhaps not related. 

Material in this monograph w^s first ass^e^led by the 
author as lecture notes for the senior seminar in^athematics 
at the University of Tennessee. . In this seminaY oneimeeting 
per week was for a lecture on the subj ect 'matter ^and another 
meeting was to permit students to present sojutions'vto - 
exercises^ Two major problems were encountered the first - 
• quarter the seminar was given. These were 'that some of the 
students had had only the required one-quarter course 'in 
matrix theory and were not sufficiently familiar with 
eigenvalues, eigenvectors and. related concepts, and that many 
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of the exercises required fortitude^ At the suggestion of - 
the UMAP ^Editor, the approach inx the present monograph is 
(1) to develop the material in terms of full rartk factoriza- 
tions and to reLegate all discussions using eigenvalues and 
eigenvectors to exercises, and (2) to include an appendix of 
hints for exercises. In. addition, it was suggested that the 
order of presentation be modified .to provide some mot nation 
Jror considering generalized * inverses before developing the 
plgebraic theory. This has been accomplished by introducing 
pe Moore-Penrose inv^ir^e of a matrix and immediately 
exploring its use in characterizing particular solutions to 
-systems of e'quations before establishing many of its alge- 
braic properties'. 

To prepare a monograph of limited length for use at the 
undergraduate level precludes giving extensive references to 
original sources. Most of the material can be found in 
texts such as Beir- Israel and Greville [2] or Rao and Mitra 
Ullj 

Every career is always influenced by colleagues. The 
author wishes to express his appreciation particularly to 
T.N.E. Greville, L.D. Pyle and R.M. Thrall for continuing 
encouragement and availability for consultation. 



Randall E. Cline 
Knoxville, Tennessee 
September 1978* 
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Introduction 



1,1 Preliminary Remarks 

\ ~- < 

The material in this monograph requires a knowledge of 

basic matrix theory available in excellent textbooks such as 

Halmos [7], Noble [ - 9] or Strang [12]. fundamental definitions 

and concepts are, used without the detailed discussion Which 

wouH be included in a self-contained work. Therefore, it may 

be he'lpful t^have a standard linear algebra textbook, for 

reference if needed. ^ * 

7 Many examples and exercises are included to .illustrate 
and compl'ement the topics discussed in the text. Ijt is recom- 
mended that every exercise be attempted. Although perhaps 
not always successful, the challenge of distinguishing among 
wharcan be assumed, what is known and' what jjiust be shown is 
an integral part of the development of the nebulous concept 
called mathematical maturity. 
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Matrix Notation \and Terminology 



JTJvroughout subsequent sections capital Latfin letters 
^ denote matrices and small Latin letters denotelcolumn vectors/ 
Unless otherwise stated, all matrices (and thus\vectors — being 
matrices having a single column) are assumed to 'hay e complex 
* . numSers 'as elements. Also, sizes of matrices j/re assumed to 
* be arbitrary, subject to corfformability in suf^s and products. 
"For example, writing A+B tacitly assumes A and B have the 
same^size, whereas AB implies that A is m by n and B is n by 
p for -some m, n and p. (Ncfte, however, that even with AB 
defined, BA is defined if and onl} if mv= p.) The special 
symbols I and 0 are used, to denote the n by n identity matrix , 
and, the m' by n null matrix, respectively, with sizes deter- 
mined by the context -*hen no subscripts are used. If it is 

important to emphasize size we will write I or 0 

r" n n 



mn 



•For any A = Ca-^ J , the conjugate transpose, of A is 
Written as A H . Thus. A H - (a^), where I... denotes the con- 
jugate of the complex scal'ar a^, and if x is a column vector 
with components x x , . . . ,x*, # then x H is the row vector ■ 

Consequently, for a real matrix (vector) the superscript "H" 
denotes transpose. 

Given vectors x and y, we write the inner product of x ' 
and y as 

v 

H n 

j . 0> x ) s x y = I H y . 

' , i=l 1 1 ' ^ 

Since only Euclideafn rfcrms will be considered, we write ||x|f 
without a subscript to mean ^ v ' 

\ ■ I x|| = +/|xTxT - */l |x.| 2 . 

< i=i 1 

NT 

To conclude this se^rtTon it is noted that there are 
certain concepts in the previously cited textbooks v^hich are ^ 

t, 9 V -2- ' 1 
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either used Implicitly or discussed in a maimer, thai does not 
emphasize their importance. for present purposes. 'Although 
sometimes slightly redundant, the decision to include such 
topics was based upon the d^ire ta stress fundamental 
, understanding. 



Exerci ses 

1.1 Let x be any m-tuple and y be any n-tuple. 

' H H 

a. Form xy and yx . 

' ' >• 

b. Suppose m » n and that neither x nor y is the zero vector. 

Prove that xy H is Hermitiap if and only if y - ax for some real 
sea I ar a. « 

1.2 Let A be any m by n matrix with rows ,w H and c0 l U mns 

Xj,. .. ,x n and let B be any n by p matrix wi th rows y H and 

columns z. z. r 4 x 

1 P , • « 

a. Prove/that the produc^AB can be written as 



ve/that t 



AB 



( V w m } 
i m 



( Zp ,w,) 



< Z n' W J 

p m 



and also as 



2 H 
.AB- Ix.y. H . ' 

b. Prove that A « 0 if and only if either A H A - 0 or AA H - 0. 

H H 

c. Show that BA A « CA A for any matrices A,B and* C implies 
■ BA H »CA H . 

* ' 

*l«3 Let A be any nor/rial matrix with eigenvalues A lf ,.,,A and ortho- 
t In 
normal eigenvectors x.,..,,x . 

I n 



*Exercises or portions of exercises designated by an asterisk assume 
a knowledge of eigenvalues and eigenvectors. 
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a. Show that A can be written as 



A * 7 A.x.x. . 

. , i t i 



b. If E. a x.x. H ,i»l n, show that E.* is Hermi t ian and~ idem- 

t - it* * t i 

potent, and that E.E . - E.E. * 0 for all M j. I 

c. Use the expression for A in 1.3a and the result of 1.3b to 
conclude that A Is Hermit Ian if and only if «|1 eigenvalues 
X. are real . 



1.3 A Rationale for Generalized Inverses 

*Given a square matrix, A, the existence of a matrix;, X, 
such that AX a I is but one of-»many equivalent necessary and , ' 
sufficient conditions that A is nonsingular. , (See Exe^clTse 
1.4.0 'in this case X ■ A * is the unique two-sided /inverse of 
A, .and x » A H is the unique solution of the linear algebraic * 
syjt.ein of equations Ax = b for every right-hand side b. Loosely 
speaking, the^ theory of generalized inverses of matrices is 
concerned with extending the concept pf an"" inverse of a square 
nonsingular matrix to singular matrices and, more- generally, 
to rectangular matrices by considering various sets of equa- 
tions which A and X may 1>e required to* satisfy. ^For this 
purpose we will use combinations of the following five^ funda- 
mental equations: s 

k k 

(1.1) A XA * A , for some positive integer.k, 

(1.2) 1 XAX » Xo 

(1.3) CAX) H '» AX,' . % *" 

(1.4) (XA) H " XA, * « ' • - 
(1.55f / AX =.XA. 

(It sWbuad be note^ that (1.1) with \l > 1 and (1 . 5) ' implicitly 
assume A and .X* are square matrices, whereas (1.1) with k = 1, 

ft , » u 

•(1.2), (1.3), and (1 .4), require only that X has the size of A . 
Also, obseriie that all of the equations clearly Ijold when A 

. '-4- 
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n is square and nonsingular, and X » a" 1 .) Given k and subsets 
of Equations (lcl)-(l.S), 'it is logical to ask whether a 
solution X exists, is it unique, 'how can it be constructed 
and what properties does.it have A These are the basic ques- 
tions to be explored in subsequen Ahapters . 

In Chapter 2 we establish the existence and uniqueness 
of a pacticular generalized inverse of any matrix A (to be -• 
called the Moore-Penrose inverse of A), and^show how this 
invterse can be. used to characterize the minimal norm or least 
squares^ solutions to. systems of equations Ax = b when A has * 
full row rank of full column, rank . . This- inverse is then 
further* explored in Chapter 3 where many .of its properties 
"a-ne derived and certain^.applications' discussed. In Chapter 4 
we consider another unique generalized inverse of square 
mtftrices A ("called the Drazi'n inverse of A) and relate this 
ihwrm to Moore-Penrose inverses. ■ The'- concluding chapter is-, 
to provideSa brief introduction to the theory of generalized " 
inverses thaV are -not unique.' 



Exercises , > 

' ,For a "y A "> •« "(A) denote the null space of A, that is, 
^ N(A) = (z|Az - OK ' • 

« a. If A is a n by n matrix, show that the s fol lowi ng, conditions 
£ are equivalent: 

% i -\ 

t i ) A is nonsingular, 

* * '/ . • \ > 

(m) ,iA|£A) contains only the riull-vector, 

(JM) Rank (A) * h, 

(•y) A has a rigjn inverse, 

(v) Ax = b has a unique solution for every right-hand 
si de b. v 

b. ■ What other equivalent statements can be added^to this list? 



1.5 .Let 



V 



1 2 i i : 




-l -i -l k' 


— 




1 0< 0 -1 




, x - 


1112 




0 J 0-1 


) 1 1 V 




0 0 1-1 
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1.6 



Show that X 



If Aj is the five* by five matrix obtained by extending ir 
the obvious manner, that is, A,. « ( a jj' wnere » , 



2, if i - j-1 , 
1 , otherwi se, 



-I 



V 



form A- . More generally, given A of this form for any 
* -1. • n 

n >_ 2, that is , 

b wi th b i ntegraf has 



n > 2* what Ls A ,? 
— n 

*P,rove that A is unimodular for a 
n 



|det A | ■ 1 , 



>s J^^r 



tegra 



d. Show that any system of equations 
an integral solution x.- 

Let x be any vector with [|x|| "T^and let k be any real number. 

. H 



a 
~b. 



Show that A « I + kxx is nonsingular for all k t -1 , 
Given the forty by forty matrix A ■ ( a jj) w ' ( tn 

f7, If i - j, 



construct A" 



I , otherwi se , 



t.' Show that A in 1.6a is an involution when k » -2. 
d. Show that A is idempotent ,when k = -T. 

■4 

*c. Show that A has one eigenvalue equal to l+k and all other 

eigenvalues equal to unity. (Hint: Consider x and any vector 
y orthogonal to x.) 

*f. Construct an orthOnormal set of eigenvectors for A^in 1.6b. 

i 

1.7 Given the following pairs of matrices, show that A and X satisfy 
(1.1) with' k - 1/ (1.2) , (1.3), and (1.4). 



X - 1/10 



0 0 
2 '-2 
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! 2 

2 

-1 \ 

0 1 

> 2' 

3 6 



»' X 



1.8 Show that the matrices 

-5 -6 >V] 

1 2 l/ 1 

2 2 .'V 2 

i*V 1 1 



1/51 



1/50 



X » 1/2 



"16 14 
15 -6 



1 3 

2 6 



-5 
2 4 
1 1 
0 0 



-7 3 
3 6 



-31 
2 2 
2 1 
0 0 



sVtisfy (1.1) with k = 2, (1.2) and (1.5). 
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" 2 
Systems of Equations 

and the 
Moore -Ffenrose Inverse 
of a Mat rix 

2.1 Zero , One or Many Solutions of Ax = b ' 

Giwsn.a linear algebraic system of m: equations -in' n 
unknowns written as Ax = b, a standard method to determine the 
.•number of solutions is to first reduce^the augmented matrix 

[A,b] to row echelon form. The numbeV of solutions is then 
character zed by relations among the number of unknowns, 
rank (A) and rank ([A,b]). In particular, Ax = b is a consis- 
tent system of equations, that is, there 1 exists at le.ast one 
solution, if and only if rank (A) = rank ([A,b]j. Moreover, 
a consistent system of equations Ax = b has a unique solution 
if and only if rank (A-) • n. On the other hand, Ax = b has no- 
exact solution when rank (A) < rank ([A,b]). It is -the 
purpose of this chapter to show how the Moore-Penrose inverse 
of A can bemused to distinguish among-these three cases and ' 
. to provide alternative forms of representations which are 
frequently' employed in each case. 

• For any matrix, A, let CS(A) denote the column space of 
A, thatMs, v - 

\ » 

-9- 
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CS(A) = (y|y*=Ax, for some vector x), 

Then Ax */b a consistent system of equations implies beCS(A)~ 
and c'onversely (which is simply another way of saying that A 
and [A,i] have the same rank). Now by definition, \ 

'„ rank (A) ■ dimension,.(CS(A)) , 

^and if fc 

N(A) = {z|Az** 0} 

denotes the null space of A* (Cf. Exercise 1.4), then we have 
/the well-known relation that N 

♦ rank (A) + dimen$ion (N(A)) = n. 

Given a consistent* system of equations Ax = b with A m by 
n of rank r, it follows, tHerefofe^ that if r = n, then A has 
full column rank and x = A^b is the unique solution, where A^ 
is any left inverse of A. The problem in this case i§ thus 
to construct A^b . 

Jlowever , when r < n, so that N(A)" Consists, of more than 
only the zero vector, then for any solution,' x/, of ,Ax = "b, 
any vector zeN(A) and any scalar = x i + az * s also a solu- 

tion of Ax = b. fc Conversely, if x^ and X2 are any pair of 
solutions of'Ax - b, and if z = x-y - *2> then Az. » Ax^ - AX2 
- b - b 3 0 so that zeN(A) . Hence all solutions to Ax = b in* 
this case can be written as ' t , 

(2.1) X = X x + J 0^2:, . - rf; 

i=»l , * 

f 

where x, is any particular solution, 2 1 , . . . t z are aj*y-T«*^ - v^, 
of vectors which form a basis of N(A) andt , . . . ,bi n / are ' Ig*^ 

arbitrary Scalars. * & f ^ 

Often the problem now is simply to characterise all solu- 
tions,. More frequently, it is to determine those solutions 
which satisfy one or more additional conditions Vas , for example, 
in linear programming where we wish tp construct) a nonnegative 



solution of Ax « b which also maximizes (c,x) w^ere c is some 
given vector and A, b and c have real element^ 

-10- 
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Given an inconsistent system of equations Ax = b, that 

is, where rank (A) < rank [A,b] so that there is no exact 
* - •re- 

solution, ,a* frequently used procedure is to construct a vector 

x,.s_ay, which is a "best 'approximate" solution by some ' ' 

criterion. Perhaps the most generally used criterion is that 

of least squares in which it is required to determine x to 

minimize ||Ax ^b|| or, equivalently , to minimi/e | |Ax -b|| 2 . 

In this case, if A has full column rank, then x = (A H A>;l'A H b 

is the least squares solution (see Exeroise 2.7). V 



Exerci ses ' 

2.1 Given the following matrices, A., and vectors, b. , determine which 
of the sets of equations A.x = b. have aVutitque solution, infi- 
nitely /tony *olut ions or no exact solutiqn ? "and^cons truct the 
unique solutions when they exist. 



(i) 






2 


7 




*-8' 






-1 


1 


2 












-3 


1_ 




J>^ 


(ii 


i) - 


"} 


- 1 


0" 










0 


3 


1 




2 




V 


5 


-2 


5 


•V 


8 






1 


0 


3 




2 


(v) 




~2 


P 


r 




-V 




V 


1 


0 


-i 




2 


• 




.3 


1 


0_ 




.-5. 



(ii) 
A. 



(iv) 
A, 



(vi) 
A, 



2 1 
1 0 

3 6 



2 1 ] 

3 1 0 
1 0 -1 



2 3 11* 
12 0 1 



H 2 

0 



2.2 For any partitioned matrix A * [B r R] with B.nonsingular, prove that 
columns of the matrix 



-b-'r 



form a. basis' of N(A) . 
2.3 Construct a basis for N(Ag) in Exercise 2.1. 
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1.4 Apply the Gram-Schmidt process to the basis in Exercise' 2.3 to 
construct an orthonormai basis of N(Ag)., 

1.5 /Show that if Zj and are any vectors which form an orthonormai 
basis of N(A^).and if all solutions of A^x » are written as 

x 



where 



Xj + (XjZj + a^^2 



-j^ro -1 1 2], 



* 2 2 

then | + |a 2 | ■ 1 for every solution such that 



=25/24. 



H H 

Show that A', A A and AA have tHe same rank for every matrix A. 

a. Given any system of equations Ax = b wi th A m by n and 

rank (£) » n,'Show by use of calculus that the least squares 
solution, x, has the form Suppose m « n? 



b. 



Construct the least squares solution of Ax = b if 



1 2 
1 0 
1 1 



, b » 



2.2 Full Rank Factorizations and the Moore-Penrose Inverse 

Given any matrix A (not necessarily squar^) , it follows 
at once that if Xfis any matrix such that A and X satisfy (1.1) 
witfc k ■ 1, (1.2), (1.3) and (J..4), then X is unique. For if 



'it* k ■ 1, 
2.1) a: 



(2.2-) AXA = A, XAX^X,* (AX) H = AX, "(XA) H * 

and if A and Y also satisfy these equations, then 



X = XAX - X(AX) H ' = XX H A H 



0CX H (AYA) H 



XX H A H (AY) H 



XAY = (XA) H Y = A H X H Y 



= (AYA) H X H Y = (YA) H A H X H Y 



= VAXAY = YAY = Y . 



Now if A has full row rank, then with X any right inverse of A, 
AX « I is Hermitian and the first two equations in (2,2) hold^. 
Dually, if A has full column rank and X is any left inverse of 
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A, XA ■ I is Hermitian and again the first 1 two equations in 
(2*^2) hold. As shown'in the following, lemma, there is a 0 
choice of X in both cases so that all four conditions hold. 

LEMMA 1: Let-' A be any matrix with full row rank or full 
column rarik. If A has full row rank, then X = A H (AA H )" 1 is 
the unique right inverse of A with XA Hermitian. If A has 
full column rank, then X = (A H A)"^A H is the unique left 

inverse .of A with AX Hermitian. 

1 j> 

V*oo'f* If A is any matrix with full row rank, AA H is non- 
singular by Exercise 2.6. Now X = A H (AA H ) _1 is a right 
inverse of A, and 

(XA) H * (a H (AA H )" 1 aJ H = A H (AA H )- 1 A = XA*. 

Thus A and X satisfy the four equations in (2.2), and X is 
unique . 



The dual relationship when A has full column rank follows 

Li * 

in an analogous manner with A A nonsingular. I 

It ^should be noted that X = A" 1 in (2.2) when A is square 
and nonsingular, and that both forms for X in Lemma 1 reduce 
to A in this case. More generally, we will see in Theorem 4 
that the unique X in (2.2) exists for every matrix A. Such 
an X is called the Moore-Penrose inverse of A and is written 
A . Thus we have from Lemma 1 the special cases: 



(2.3) 



A = 



A H (AA H )" 1 , if A has full row rank, 
ft 

(A H A) -1 A H , if A has full column rank. 



Example 2.1 
* If A ■ - 



then*(AA H )"^ 



'2 


1 




"2 -1" 


\ 


2_ 


3 

' 6- 


-1 2 _ 



j^nd so 



" 7 



UC 



"I" 

2 
1 
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Exampl 



pl<T2.2 



I*f y is the column vector- 



y = 



1 

2+i 

o 

-3i 



then 



1? 



[1 2-i 




Note in general that t'Ef Woore-Penrose _inversfe of any nonzero 
% row or column vector itfjimply the conjugate transpose ^of the 
vector multiplied by ty^peciprocal of the square of its 
length. / 

-"Let us* next £on^jjdj|r the* geometry, of solving systems of- 
equations in terms °£f.^* for the special cases in (2^3). 
Given any, system of e)q^tions Ax - b* and any matrix, tf", such, 
that AXA = A,*it fallow;^ at once that the system is consis- 
tent^A and only 'if? ' m 

(2.4) AXb - b, \ Jj' 

For if (2.4) holds,' the^x ■ Xb is "a solution. Conversely, 
if Ax = b is consistent ^multiplying j?ach- side oh the left by 
AX gives Ax v ■ AXAx ■ AXb* $b that (2.4) -follows. Suppose now, 
that Ax * b is a system of ! m equations in n unknowns where. A 
has full row rank. Then Ax '« J> is always consistent since ^ 
(2*4) holds with X any right^ inverse o£ 'A, and we have from 
(2,1) that all solutions can be writfen as .' 

(2.5) x » Xb * 2y l * s \ 
• . \\ ' || > , s < " 

with Z any matrix with^ n-nfclcol|umns which forrif a basis of N(A) 

and y ajj arbitrary vector,^ Taking X to, be the right inverse- 
A + in thi,s case gives fheoyem ^ , 

THfeOREM 2: For' any system|^|^quations Ax 
full rbw rank, jc * A + b is $mjti ini^ue » solution with ||x| ! 
minimal . 



b where A has 
V 
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\ 



o Proof: 



With -X »-A + in*(2.50, ' * o „ " 

llxjf- 2 = (x,x) = (AVzy,A*b+Zy) ^ , / . 

\ - CA + b,A + b) * (Zy,Zy) = | | A + bT| *V j | iZ ^| 2 



since 



\aTM)0 = (A H (M H ); 1 b,2^) = ((AA H )' 1 b,AZy) * 0. 
' IM| 2 > l|A + b|'| 2 , 



where equality Jmlds if and only if Zy. - <^.\l r A 
Example 2.3 . . 0 / > * * 

If A is the'matrix in Example 2.1 and^b * ^ j then 



x = A*b -J 



14 
-13 
1 . 



,2 _ 122 



is the minimal norm solution oj Ax = b witK. J |x|| = % 

It was not.ed in Section 2.1 that thej. least squares splu- 
tion of an inconsistent system o£ equations Ax = *b when A has* 
full column rank is^x'= (A H A) "^b . From** (2. have, 
'therefore, that x =» & + b is the least squares solution in this 
case. Although this result can be established by use'of 
calcjulus (Exercise 2 . 7) , 'the,, following derivation in terms'.'of 
norms is more direct. . ' \* „ . 

THEOREM 3: <£br any system* of equatj^ons Ax * b where A, has 
^Sfull column rartk> x = ^*b is "the* unique yector with | |b-Ax||? 
, minimal. *' • * « • « * • . r »f 

Proyf - j If A is square If mo* nvand Ax * *b is consistent, 
then with a\ » Y^AJ^A^ left inverse of' X\and AA + b » b, the 
vector x = A + b is*t£ e\unique solution with, ||t>-Ax|| 2 * 0. On 
the other hand, if m ">\h *and Ax = b i* "inconsistent, 

| | b-Ax |"| 2 - H(I-AA't)b-A(x-A*b)||^ ' . * 

» ||b-AA + b'| | 2 + HA(x-A + b)i1 2 ^ v 

. -15-- . 



since A (I-AA ) » 0. Hence | |b-Ax| | 2 > | |b-AA*.bJ | 2 where' 
equality holds if and only if | | A(x-A + b) | | 2 = 0. But A with 
full column rank implies | |Ay| | 2 > 0 for any vector y t 0, 
in particular for y * x - A + b. | 

Example 2.4 

If 





' 2 


-V 




"1 ' 


A = 


1 


1 


, b « 


2^ 




-1 






3 m 



then 



and 



AA + b = 



1 

TT 



A b = 



1 

TT 



is the least squares solution of Ax = b with ||b-Ax|| 2 = -yji 
minimal . 



Havirfg established jj£ for , the special cases in Lemma 1, 
it remains to establish existence for the general case of an 
arbitrary matrix A. for this purpose* we firs^- require a 
definition. 

~ DEFINITION 1; Any product EFG with E m by rA 
F r bry 'r and G n by n is called a full rank 
factorization if'each of the matrices E, F and G 
fcas rank r, ^ - 

The importance of Definition 1 is that any nonnull matrix can 

be expressed in terms of full rank factorizations s and that 

the Moore-Penrose inverse of such a product *is the product of 
the corresponding inverse rfi revers"e ordef . 

To construct a full rank -factorization of a iionnull * 
matrix, let A be any m by n matrix with rank r. Designate 
columns 'Of A as a^...^. Then A witn ran* r implies that 
there exists at least one set^rrf r colujrm? of A which are 



I 



linearly indepen4ent. Let J * {j^ 

indices for which a; ... 

' J l 

let E b'e the m by-.r matrix 



\ 

,3 r } be any set of 



,aj^ are linearly independent, and 



[a 



•v 



If r a n, 'then A ■ E is a trivial full rank factorization 
(with F - G * I). Suppose r < n. Then for every column 
*y)tJy there is a column vector y^ , say, such that a^ 3 Eyj t< 
Now form the r by n matrix, G, with .columns g£,...,g n as . * 
follows: Let 



e i' 



if j*J, 

if j = j^J, 



where- ,i=l , , 
we then have 

EG = [a 



,r, denote unit vectors. For this matrix G 



1' 



Moreover, since the columns e,,...^ of G form a r by r 

* ' ' ' r » « ■* 

identity matrix, rank (G) - r, and with rank, (E) = r by*con- 

structioh, A » EG is a full rank factorization (with F = I). 

.That a full rank factorization A = EFG is not unique is 
• apparent by observing that ^fj^M and N are any Jionsingular, 
matrij|s, tlTen A^ EM(M" 1 fN)N 1 G is also a fulA rank factori- 
zation. The following example illustrates four full rank 
' factorizations of a given matrix, A, where F *= J in each case. 



Example Z.5 
Let 

s A * 

Then 

4 

A * 



2 
1 

-1 



4 2, 



* 6 
-1 
-15 



3 " 
-4 



2 
1 

r-i 



-2 



9 

ERIC 



•17- 



24 



4 6 
1 -1 
-*5 -15 



'4/5 
-1/5 



3/5 1 
•2/5 0 



.7/5 0 
-'3/5 1 



-1 1 

n 0, 



0 -7] 

1 3} 



Using full rank factorization, the existence of the Moore- 
Penrose inverse of any matrix follows at once. The following 
theorem, stated in the fo,rm rediscovered by Penrose [10] but 
originally established by^ Moore [8], is fundamental %o the 
theory of generalized inverses of matrices. ^[ t 

THEOREM 4: For any matrix, A, the four equations 



AXA - A, XAX = X, r UX) H = ,AX, (XA}" 



XA 



have a unique solution X = 4*. If a 



°mn is the m by n nul1 
matrix, A T = 0 nm . If A is not the null matrix, then for any 



G*F"V. 



ull fank factorization EFG of A, A H 

*rpof: Uniqueness in every case follows from the remarks 
after (2.2). * N 

If A is 



If A * = °mn' then XAX 



X implies X = A 



0 



n6t the^null matrix, then for any full rank factorization 
A > EFG it follows by definition that E has full column rank, 



(E H E) _1 E H 

G 11 (GG n )~% by (2.3), with E + a left inverse of E and 



F-is nonsingular and G has full r|>w rank. Thus.E 



and G 

G a right inverse of G. Then if X 
AX = EE are Hermitian, by Lemma 1. 
XAX' ^ X, so that X = A + . I 



GVV, 



XA = G G and 
Moreover, AXA * A and 



It should be noted that although- the existence of a fuld 
rank factorization A », EG has been established for any non- f 
null matrix A, this does not provide a systematic computatiqnal 
procedure for constructing a factorization. Such a procedure 
will be developed in Exercise 3.3, however, after we have - 
considered the relationship between A + and the Moore-Penros e 
inverse of matrices obtained by permuting rows or columns or 
both rows and columns of A. Observe, moreover, that if Ax = b 
is any system of equations with A = EG a full rank factoriza- 
tion, and if y » Gx, then y.= E + b is the least squares- 
solution to Ey = b, by Theorem 3. Now the system of equations 

r , -18- 
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•Gx - E b is always consistent and has minimal norm solution 
* X - G E b, by-TBeorem 2. Consequently we can comb,<£e the >' 
, results of Theorems, 2 and 3 by saying that x = A + b is the ' 

least squares solution of Ax - b with |jx|| 2 .minimal. 
^ Although of mathematical interest (see, for example, Exercises 
3.12 and 3.13), most practical applications of' least squares, 
require that problems be formulated in such a way that the 
matrix A has full column rank. \ 

Exerc 1 ses - - 

2:8 Show that X] ■ A 6 + b in Exercise 2.5. c 
2.9 Show thatr If A Is any nonsin 9 ular matrix,; then 
"A H (AA H + BB H f 



1 [A,B] + 



fl H (AA H + BB H ) H 



2.10 Let u bj the col uiW- vector withn elements each equal-td unity. 
Show that 



(n+l)l-uu 



2.11 a. Gi^any real numbers b r ...,b n , show that all solutions to 
the equations v »" 



x f + x 



n+1 



can be written as. 



5 :■_!_ 

3 I ri+l 



+ a, i » 1 



- . , n , 



and 



1 

A n+1 " 'nTf 



where a Is arbi trary . 
For what'cholce o^a can we 




e the additiona-l condition 



Show that when the condition In 2.11b Is Imposed, x . 
becomes simply the mean of b.,...*,b n> that is, 



*n+l 



1 I b.\ 
n i-l ' 



ft 

tie e 



d. Show that the problem of solving tje equations in 2.11a, 
subject to the conditions in 2.11b, can be formulated 
equivalently as a system of equations Ax « b with the n+l 
by n+l matrix, A Hermitian and nonsingular. 

2.12 Given any real numbers bj,*..,b n , show that the mean is the least 
^"squares solution to the, equations ^ 



i 



. ,n. 



H 



2.13 If Ax « b is any system of equations with A * uv, a matrix of rank 
ope, sbdw that 



!u|| 2 ||vl| 2 



is the least squares solution with minimal norm. 
2".}^ Let Ax * b be any consistent system of equations and let 



,z be any -set of vectors which form an orthbnormal basis 
. n-r 



I* 

of N(*)» where rank (A) » r. Show that if x is any solution of 
Ax » b„ 



A + b » x 



n-r 

- la.z. 

& 1 ■ 



with cij ■ ^x,z.) , i - I , . 



. ,n-r . 



2.15 (Continuation): Let A be any m by n matrix with full row rank, and 
let Z be any 0 by n-m matrix whose columns form an o?fe^onormal 
basis of ( M(A).- Prove that if X is any right inverse of A, 



22 H X. 



2.16 Use the results of Exercises 2.4 and 2.15 to cdnstruct A^ starting 
with the right inverse 



2 -3 

^1 2 

0 0 

0 0 
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2.3 Some Geometric Illustrations 

'In this section we illustrate the geometry of^ Theorems 2 
and 3 with some diagrams: 

Consider* a single equation in three real variables of 
the form 



(2.6) 



a ll x l + a J2 x 2 *' a 13 x 3 "*r 



Then it is well known that all vectors x H = [x^x^Xj] which 
satisfy (2.6) is a plane ? 1 (b 1 ), as shown in Figure 1, Now 



the plane P 1 (0) is 




Figure I. The plane P,(b,) and solution x. 



parallel to P 1 (b 1 ), and consists of all solutions ^ 
2 * f 2 l» z 2 ,2 3^ t0 t ^ ie homogeneous equation.' 

(2.7) a nZl + a 12 z 2 + a 13 2 3 - 0. 

Then if b^j 0, all solutions xeP 1 (b^) can be written as 
x * x + 2 for some 2eP 1 (0), and conversely, as shown in Figure 
2, (Clearly, this is the geometric interpretation of (2.1) % 
for a single equation in three unknowns with two vectors 
. -required to span P^O}.) If we now let a^ * [a^ ,a 12 , a 13 l , 
so tha^ (2.6)*can be written as a 1 H x * b 1 , Theorem 2 implies 

♦ '< 
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Figure 2. Pjtbj), P } {0), x, z and x. 

that the solution of the form x = a 1 H+ b 1 is the point on 
P^bj) with minimal distance- from the origin. Also, since 
the vector x is perpendicular to the planes P 1 (0) > P 1 (b 1 ) , 
||x|| is the distance between P^O) and P 1 (b 1 ). The repre- . 
sentation of any solution x as x = a 1 H+ b 1 + c^z, corresponding 
to* (2.1) in this case, is illustrated in Figure 3. 




Figure 3.* The representation x p a ( bj + ctjZ. 
* -22- 



Suppose next that we consider (2.6) and a second equation 



(2.8) 



a 21 x l 



a 22 x 2 



a 23 x 3 \ b 2' 



Let the plane of solutions of (2.8)\ be designated as P 2 (b 2 >. 
Then it' f Alow* that either the plates P^) and P 2 (b 2 ) coin- 

, cide, or they a*re parallel and distinct, or they intersect in 
a straight line. In the first case, when P 1 (b 1 ) and P 2 (b 2 ) ' 
coincide, the equation in (2.8) is a multiple of (2.f) and 
any point satisfying one equation also satisfies the other. 
On the other hand, when P^bj) and P 2 (b 2 ) are parallel and 
distinct, there is no exact solution. Finally, when P 1 (b 1 ) 
and P 2 (b 2 ) intersect in a straight line l u , say, that is, 

.*U = P iC b i> H P 2 f b 2 } ' then an y P oint on £ 12 satisfies both - 

;;(2.6) and (2.8). Observe, moreover, that with 





!ll a 12 a 13 










A = 




, b = 


ft 


* 


/ 




a 21 a 22 a 23 











the point on l 12 with minimal distance from the origin is 
x = A b. This last case is illustrated in Figure 4, where 
is a "translation" of the subspace N(A) of the form 

P!(0)np 2 (O). 




Figure VV' P 2 (b 2 ) » £ 12 and A+b ' 
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The extension to three or more equations is now obvious: 
Given a third equation 

(2.9) a 31 x 1 + a 32 x 2 + a^ = b 3 , u 

let P3(b 3 ) be the associated plane of solutions. Then assum- 
ing the planes P 1 ( b 1 ) and ^1^7} do no ? c0in 94$ e oxjnfe not! * 
parallel a,nd distinct, that is, they intersect in the line i, 7 

t H 

as shown ;in Figure 4, the existence o£«a vector x = (x^,^*^) 
satisfying (2.6), (2.8) and (2.9) is determined by the condi- 
tions that either P 3 (bj) contains the* line & 12 , or P 3^ b 3) an< * 
* 12 are parallel and distinct, or P^lTb^ and * 12 intersect in 
a single point. (The reader is ur^edr to construct figures to ft 
illustrate these cases. An illust&tjon of three different 
planes containing the same line may also be found in Figure 
5.) For m ^ 4 equations, similar considerations as to the 
intersections of planes P v (b,) and lines I- . = P. (b.) OR (b.) 
agadn hold, but diagrams become exceedingly difficult to 
visualize. 

For any system of equations Ax = b let y = AA + b, that is, 
y is the perpendicular projection, of b onto CS(A), the column 
space of A. Then it follows from (2M) that Ax 3 y is 
always a consistent system of equations, and from Theorem 2 
that A + y = A + (AA + )b - A + b is the minimal norm solution. More- 
over, we have from Theorem 3 that if Ax = b is inconsistent, * 
then 

||Ax-b|| 2 . = ||y-b|| 2 = ||AA + b-b|| 2 

.minimal. Thus, ,the minimal norm solution A + y of Ax = y 
2 

minimizes | |y-b | | . M 

Consider an inconsistent system of, say, three equations 

two unknowns, Ax = b, and suppose rank (A) = 2, Let 
+ * H H H 

AA b ihave components y^)^')^' and let a i » a 2 ,a 3 de sig- 

nate rows of A. Now if P^{y) is the plajie of all solutions 

H " ' * 

of a^ x * y^ t i ■ 1,2,3; then 'the set of all solutions of 

Ax * y is the line a P i(yi^ ^ p 2^ y 2^ as snown » in Figure 5a, 
A*y ■ A + b is the. point on of minimal norm and |(y-b|[ 2 is 
mini^a-1 



9 
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as shown in Figure 5b. 
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(I-AA + )b 



(a) . (b) 

Figure' 5- (a) Solutions of Ax » y where y * AA + b. 

(b) The vectors b, y and &-AA + )b. 



To conclude this section we remark that since 
b = AA*b + (I-AA + )b * 
is an orthogonal decomposition of any vector b with 

j|b|| 2 = ||AA + b|! 2 ♦ ||(I-AA + )b|| 2 , 
then the ratio 



(2.10) 



||<I-AA + )b|| 2 ~ 



provides a measure of inconsistency of the system Ax « b. In 
particular, <J> * 0 implies b is .orthogonal to CS(A), whereas 
large values of <J> imply that b is nearly contained in CS(A), 
that is, | | (I-AA + )b| | 2 is relatively small. (For statistical 
applications [1] (4] [5], the values | |b | | 2 , | |AA*b | | 2 and 
| |.[I-AA + )bf| 2 are frequently referred to as TSS [Total sum of 
squares!, SSR [Sum of squares due to regression] ancl S$E (Sum 
of squares due to error], respectively. Under ^certain general 
assumptions, particular multiples of <J> can be shown to have 
distributions which can be used in tests of significance , ) 
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Although the statistical theory of linear regression models 
is not germane to the present considerations, formation of <j> 
pi (2.10) can provide insight into the inconsistency of a 
system of equations Ax * b. (See Exercise 2.18.) 

Exerc ises 

2.17 Use the techniques of solid-analytic geometry to prove that the 

lines i ]2 » P,(b 1 )nP 2 (b 2 ), b } t 0 ahd b 2 * 0 and I ]2 » P ] {0)(]? 2 (0) 
are parallej. In addition, show by similar methods that if 



P,(b.) = {> 



b., a. - (a ir a. 2 ,a. 3 )}, 



theo 



lla.jSb.ll 2 «min||x|| 2 
xeP.(b.). 



2.18 Given any points (x.,y.), i =■ 0,1,. ..,n, i-n the (x,y) plane with 
x 0'"' ,x n d ' stinct > i s well known that there is a unique 
interpolating polynomial P (x) of degree < n Cthat is, P (x.) * y. 
for all i = 0,...,n), and if 



a n + a,x + . . . + a x 
0 1 n 



thefo a Q ,...,a n can be determined by solving the system of equa- 
t ions M ■ y Where * 



» y 



Now any matrix, A, with this form Is called a Vandermonde matrix, 
and it s can be shown that ' 

det(A> - .n (x.-x.) , 



Kj 



! 1 J 
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Thus, with x 0 ,...,x^ distinct, A is nonsingular, and if A k denotes^ 
the submatrix consisting of the first k columns of A, k * l,2,...,n 
then A^ has full column rank for every k. 

For k£n, the least squares polynomial approximation of 

degree k to the points (x. y.), i = 0,...,n, is defined to be that 
* » * 

polynomial 



P k (x) 



V + V + •••-V 



which minimizes 



r«o 1 k 1 

Show that the coefficients a Q ,...,a k of the least squares.; 
polynomial approxi,mation of degree k are elements of the 
vector a, where 



b. 



Show that with TSS * £ y * then SSR = I |A. .A. , + y 
1*0 "v 1 k 1 



Given the data 



X. 
Y 


t -1 


0 


1 


1 


y i \ 


-5 


-k 


-3 


10' 



construct the best linear, quadratic and cubic least squares 
approximations. For ^ach case determine SSR and SSE. What 
* conclusions can you 4 d raw from the data available? 

2»4 Miscellaneous Exercises 
2.|9 Let A, Z ] and Z^Jbe any ma'tri ces. 

Prove that a solution, X, to the equations XAX - X, AX « Z } 



and XA - Z 2 , if it exists, is unique. 

For what choices of Z } and Z 2 is X a generalized inverse of A? 



<i. 20 Verify the following steps In the original Penrose proof of the 
'-, existence of X in (2.2): 
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a. . The equations of XAX ■ X and (AX) H » AX are equivalent to 

the single equations XX H A H » X. Dual 1y, AXA - A and 

(XA) n « XA are equivalent to the single Equation XAA H w A H . 

b. ' If there exists a matrix sVatlsfylng BA H AA H - A H , then 

X » BA H is a solution of the equations X,X H A H « X and XAA H - A H 



The matrices A H A, (A H A) 2 , (A H A) 3 are not all linearly 

independent, so that there exists scajars dj d^ not all 

zero, for which 

' djA H A + d % 2 (A H A) 2 + ... + d k (A H A) - 0. 

(Note that if A has n columns, k < n 2 +l . Why*?) 

Let d s be the first nonzero scBlar in the matrix polynomial 
in 2.20c, and tet 




Then B(A H A) S+1 . - 



e. The matrix B also satisfies BA H AA H - A H . 

2.21 |_e* A and X be any matrices such that AXA • A. Show that, if Ax ■ » 
♦is a cons is tenT system of equations* then all solutions xan be 

« written as 

x - Xb + (i-XA)y » 

where y is arbitrary. (Note, in particular, that this expression^ 
is equivalent to the form for x in (2.5) since columns of I-XA 
forma basis for N(A) . Why?) 

2.22 (Continuation): Prove, more generally, that AWC ■ B Is a consistent 
system of equations if and only if AA + BC + C - B, In which case all 
solutions can be written as 

W - A + BC + + Y - A + AYCC + , V. 

where Y Is arbi trary . 
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More on 

- . i • t * 

Mc^re-Peniose Inverses 



3.1 Basil: Properties of A* 

The various properties of A**discusM in thtS section 
are fundamental to the theory of Moore-^nrose' inverses. In 
■any pases, proofs simply require verification that the 
defining equations in (2. 2 f are satisfied for A and some ' ; 
particular matrix X. Having iliustratJdWs proof technique 
m a number of cases, we wUl leave the Wining similar" 
arguments as exercises. 



LEMMA 5: Let A be any m by m matrix. 



Then 



(a) 

on 

CO 
(d) 
(e) 

0 



A mbyji implies A + nbym; 



mn 



implies A + * 0 



nnr 



..++ 



A; 



A + - (A H A)V . A H (AA H > + ; 



CA H A) + 



A + A H+ ; 



4 



(g) (aA) + - <x + A* for any scalar a, where 



♦ 

a 



l/a, if a + 0, ^ 
0, if a ■ 0; 

(h) If U and V are -unitary., matrices, (UAV) + - V^U* 1 ; 

n H S ♦ 

(i) If A * T A. where A. A. ■ 0 whenever i f j, A * \ A i ; 

i-1 i i-1 

(j) If A is normal, A + A ■ AA + ; 

(k) A, A + , A* A and AA + all have rank equal to trace (A + A) . 

Proof: Properties (a) and (b) have been noted previously in 
Section 1.3 and Theorem 4, respectively. The relations in 
(c) and (d) follow by observing that' there is complete 
^duality in the roles of A and X in the defining equations. 

To establish the first expression for a£ in (e) , let 
X * (A H A)*A H . Then XA » (A H A) + A H A is Herraitiaa, and also 
AX »*A(A H A)*A H by use of (d) . Moreover, XAX - X and „ m 

AXA * A(X H A) + A H A - A H+ A H A(A H A)tA H A - A H+ A H A - A. 

The second expression in (e) follows by ^similar type of 
argument, as do the expressions in (g) and (h). 

To prove (f) we have A H+ ■ A(A H A) + by (d> and (e)« Then 

A*A H * - (A H A) + A H A(A H A) + - (A H A)V 

* H 
To prove (i) , observe first that A i A^ ■ 0 implies 

A i +A j - A i W A j ■ 0 . 

and also A. + A. « 0 since * 

3 1 * < 

H ■ f/ 

A j i " - 

Now we can again show that A and A* satisfy ttie defining 
equation. 

That (j) holds follows by use ^f J[e) to write 



VA - CAVA H A - (AA H )*AA H « A H .*A H - (AA*) H - AA + . 
-30- 
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To show that A, A + , A + A and AA + all have the same rank, 
we can apply the fact that the rank of a product of matrices 
never exceeds the rank of any factor to the equations 
AA + A = A and A + AA* ; £ A V Then rank (A) = trace (A + A) holds 
since rank (E) = trace (E) for any idempotent matrix E [7]. | 

Observe in Lemma 5(e) that these expressions for A* 
redtn^to the expressions in (2.3) whenever A ha s ^HrT6w~~ 
rank or>Sq^coliijnn rank. Moreover rela- 
tionship (EjT<^^ rank factorizations 
EG, by Theorem 4, als 
by Lemma 5(f) . 
the relation (BA) 
A and B. 

Example 3.1 
Let 

n 

A * 



H 

^L? or A A wnere A is any matrix, 
The following exampl^shows , however, that 



AV 



need not .hold for" arbitrary matrices 



1 -1 
1 -1 



Then 



BA 



1 0 
0 0 



- (BA) + , 



since BA is Hermitian and idempotent. Also, we have 

[ 



B + « B^BB")' 1 1/2 



' 1 

1 1 
-1 -1 



so that 



2 -2 
-2 3 



= 1/2 




AV 



\ V" 1 ] 



) (BA) + . 



Let A be any m by n matrix with calumns a 1 ,...,a ,' and 
let Q designate the permutation matrix obtained by permuting 



columns of I in any ^orjjer {j 



1" 



Then 
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AQ ■ fa ,.\,a. ]. 

\ 

H H 

Jjv a similar manner, if w 1 , . . .w m designate the rows of A, 
and if P the permutation matrix obtained by pefftuting 
rows of I in any order {ij,... f i m }, then 



PA 



Combining these observations it follows, therefore, that if' 
A is any m by n matrix formed by permuting rows of A or col- 
umns of A or both rows and also columns of A in. any manner, 
then A =.PAQ for some permutation matrices P and Q. Moreover., 
since P and Q are ifnitary matrices, % 

r = q h a + p h , ' . ;■; ' - 

by Lemma 5(h), and thus 

+ • - + 

A„ « QA 

In other words, A + can he obtained by permuting rows and/or 
columns of A?. 1 

Example 3.2 A 
•Construct B + if 



B 



1 1 0 
1 0 1 



Since B in this case can be written as 



TO ll [~1 0 1 

B = PA v I oj [o i i 



where A is the matrix in Example 2.1, then with P and Q 
Hermit ian 
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B + - Q H A + P H ^ 1/3 



'0 


0 1" 




' 2 


-r 








" i 


1" 


0 


1 0 




-1 


i 




- 1 °. 


2 


-1 


1 


0 0. 




1 






-1 


2_ 



* (It should be noted that B can be written" alternately as 
B s AQ 1 s 



Of 



*1 0 1 
0 1 1 



0 10 

JO 0 1 

,1 0 0 

and so we have also B + ■ (J^A*.) 

Furthej applications of full rank factorizations anf of 
permuted matrices PAQ in the computation of A + will be 
illustrated in the exercises at the end of this section. We 
turn now to a\ somewhat different method for -computing A*. 
This proredure^ essentially provides a method for constructing 
the Moore-Penrose inverse of any matrix with k columns, given 
that the Moore-Penrose inverse of -the sub^trix consisting oiA ^ . 
the first k-1 columns is known. * - $ f 

For any k > 2, let A k denote the matrix with k columns, 
a l***" ,a k' ^ Tnen A i( can be written in partitioned form as 
A k = [A k-l' a k ] ' Assuming A ]( ; 1 + is knowH, -A k can be formed* 
using the fortnulas^in Theorem 6. * 

THEOREM 6: For any matrix A k = f^ k „ 1 »a k 3, let 

.A = 0 * VlVl>k 
an<V"l»t H k 



; ^ Then 
(3.1) 



H. hO* + 
Y k " V A k-l A k-1 V 



Vi + - A k-il a k b k 



where 



fc k + , ifc^.,0, 
-33- * 
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Proof: Since c k is a column vector, the two cases c k f 0 
^ and c k s 0 are exhaustive. 

/ Let X designate the right-hand side of (3.1). Then to 
establish the representation for A k + requires only that we 
show' the defining equations' in (2.2) are satisfied by A k 
and' X for the two forms of b^, n 

Forming ££X and XA k gives v ^ 
• (5.2) A k X - \_ x ^. x * ♦ c k b k> 



by definition of c k , and 



(3.3) 



XA, 



A k-1 Vl'Vl , a k b k*k-l A k-1 "k^'W 



' b k A k-l 



b k 3 k 



Continuing, using (3.2) gives 



V 



(,3.4) 
and 

(•3.5) 



k^k " [ A k-i + c k b k A k-r A k-i A k-i +a k *" c k b k a kJ. 



XA k X 



k- T a k b k A k- i A k-i + 'V i +a k b k c k b k 
" b k A k-l A k-l + + c k b k c k 



J** 



nd b k ■ c k . Their 



since A k ^ c k ■ 0. 

Suppose now tha 

; . A k X " A krl A k-l + + c k c k + -> 

in (3.2) is Htermitian. Also, with c k + c k » 1, and- since 
A + k-1 e c k » 0 implies c k H A k-1 ^ + » '0- so that c k + A k-1 - 0 and 



thus c k a k ■ 1 , then 



XA,. 



Vi A k-i 0 
a l 



in (3'. 3) is Hermitian. Moreover, 
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■ [v-i»ViViV c k] ■ [ A k-i> a k] - \ 

in -{3.4), and ' • * . -r 



XA k X - 



V 1+ -Vi +a k b \ 



. in (3.S). Having s^hown that the defining equations hold, 1 
then X » A k + in (3.1) when c k f 0. 

"■. Suppose c k - 0 and b k = (1^)"^%.^^ ^: Then 



in (3.2) is Hermitian. In this case,, with 

Y .k , a k A k-1 A k.-1 a k 
a nonnegative real number and- 

we have also 



XA, 



A k-i\-i-(^ k )'\.iV k \.i Ht 




< 14 VV\-1 



H+ 



i-dn k ) 



in (3.3) Hermitian. Furthermore, with b^.^ * « b k and, 
since c k - 0 implies VlS^H - a fc , * 



A k^k " A k 



.in (3.4) and XA k X = X in (3.5). Thus, wh^ Cfc = 0 it has been 
■ . shown again that A k and X satisfythe defining equations for 

• the given form for b, . I 
k 

4 

That the formulas in Theorem 6 can be used not only 
directly to construct a/, assuming A^/ is known,- but also • 
recursively to form A + for any matrix A is easily seen: Let 
A be any matrix with n columns a. a n , and for k = 1... ' n 
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let designate the submatrix consisting of the first k 
columns of A. Now A 1 * ■ a 1 * follows directly from Lemma 5(b) 
or (e), and if n > 2, A2*»*"»*h* " A+ can be *°*»ed 
sequentially using Theorem 6. 



Ex amp 1 e 3.3 
Let 



'2 1 0' 
0 2-4 
-1 1 -3 



Then with A 1 » a^ f 



i x - 1/5(2 0. -1J, . 



A x *a 2 - 1/5 



and 


"f 




' 2" 




"3- 


' c 2 " a 2 _A l (A i +a 2 ) " 


2 


- 1/5 


0 


■ i/s 


10 




l m 




-l m 




.6 



tence 



and so 



b 2 ' e 2 " 1/2913 10 6] 



1/S[2 0 -1] - 1/14513 10 6] 

l/29[3 10 6] 
4 - 



[55 


-10 


-35] 


- 1/29 


"11 -2 


-1 


Lis, 


50 


30J 




.3 10 


6 a 



Continuing, 



A 2 a 3 - 1/29 



29' 
-58 



••[.;] 



and 



a 3 -A 2 (A 2 a 3 ) - « 3 - a 3 
Thus, with Y3 " 5 «nd- 



•36' 
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a 3 A 2 JW 2 " ( A 2 a 3^ A 2 s ^"l* -22 -19], 



we have 



so that 



b 3 » 1/174(5 -22 -19] 



1/29 



"11 -2 


-7" 




5 -22 


-19" 






- X l/174 






_ 3 10 


6 




•10 44 


38 



1/174 



1/174[5 -22 -19] 



'61 10 -23' 
2ST 16 • -2 
5 -22 -19 



As will be' indicated in Exercise 3.7, there is a converge 
of Theorem '6 which, can be used to construct A Jc _ 1 + , given 
[A k : l' a k ] • Combining Theorem 6 and its converse thus provides 
a technique for constructing the Moore-Penrose inverse of a 
matrix^ I, say, starting from any matrix, A, of the same size 
with A known. (For practical purposes, however, X and A 
should differ in a small number of columns. 



Exercises * % 

3.1 Let A be any matrix with columns ■|..-..« ||f and let A + have rows 
W 1 '••' ,w n 4 , Prove that ,f K tenotts any subset of the Indices 
1,...,n such that aj - 0, then w ( H - 0 for all leK. 

3.2 Let A be any mbyn matrix with rank r, 0 < r < mln(m,n). 

a. Prove that there exist permutation matrices, P and Q, such 
that A • PAQ has the parti t toned form . 



with V rbyr and nonslngular. 
Show that 1 - YV^X. 
Construct A + . 
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AA 



3-3 



3.* 



(Cont I nuat Ion) 
is upper t 



;lon): 
: r tangu l\r 



matrix [U,V] Is ca I fed upper trapezoidal if U 
and nonsingular; a matrix, B, ts called lower 



trapezoidal Pf B Is upper trapezoidal. 

a. ' } Show that any matrix, A, in Exercise 3.2 has a full rank 
factorization A « EG with E lower trapezoidal and G upper 
trapezoidal. (Such a factorization A * EG is called a 

trapezoidal deoompoeition of A. ) 

Construct 4 trapezoidal decomposition of some matrix, A, 
obtained from 



-I 

0 
I ' 



Hint: Stact with 



-I 

* 
* 



where the asterisk denotes elements yet to be determined,^ 
and proceed to construct a P and Q, if necessary, so^that 
the elements e 22 and g 22 of p E and GQ, respectively, are 
both nonzero. (Note that if this is not possible, the 
factorization is complete.) Now compute -the remaining ele- 
ments In the second column of PE and the second row of GQ, 
and continue. 

• * + r - ' 

c. Compute A . v 

Let A » [B,R] be any m by n upper trapezoidal matrix with n > m + 2 
and let Zj be any nonnull vector in N(A). Show that Gaussian 

I imi nation, together with a permutation matrix, Q, can be used to 
reduce the matrix 



toan_upper trapezoidal matrix S, say. Prove now that if z 2 is 

any vector such that Sz 0 « 0, then Q H z 0 eN(A) and (z. ,Q H zJ « 0. 

*> * 12 
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3.5 
3.6 



Apply the procedure In Exercise" 3. 4 to construct an orthonormal 



basls^for. N(A) In Exercise 3,3 

Show that the-two forms for I\.p« k 'l + In Theorem 6 can be written 
In a >frigle expression as * « ' 



JVi-V + 



+ j H 

k-t Vk 



\_,-A, ,Xd 



where 



3.7 



G, 



H 



wl th d^ a row vector 



Show that 



Vi 



G k-i«- d k d k + )- if W 



b. Construct A If 



A 



10-10' 

2 10 2 

3 0 14 



3.8 



Hint: See Exercise 3.3c, 



For «ny product AS let B, - A + AB and A, - AB ) B ) + . Then 



'* — ) + 
Theorem k when AB is a full rank factorization? 



(AB) T - (AjBj)' 1 ' - B^A^*. Why does this expression reduce 



to 



*3.9 a. Use Exercise 1.3, to prove that If A Is any normal matrix, 



c . 



where indicates that the sum is taken over indices i 
with eigenvalues Aj 0, 

Prove that if A is normai , (A n ) + - (A + ) n for ail n > 1. 
If \. - 21-2 , i - 1 ,2,3, and 







~2 




' r 


0 


, x 2 - i/3 


t 


1 

x « 




: \ 




2 


3 3fi 





construct the Moore-Penrose Inverse of the matrix, A, for 
which Ax. - XjX. , I - 1,2,3. 



3.2 Applications with Matrices of Special Structure 

For nfcany applications of mathematics it is xequired to 
solve systems of equations Ax ■ b in^which A or b or both A 
and b have some special structure resulting from the physical 
considerations of the particular problem. In some cases this 
special structure is such that we can obtain information con- 
cerning the sex of all solutions. For example, the explicit 
form for all solution^ of the equations 



x i + x A*i 



1 n, 



given in Exercise V • H • was obtained using the Moore-Penrose 
inverse of the matmx [I,uJ from Exercise 2.10 where u is 
the n-tuple with each element equal to unity. In tnis section, 
we introduce the concept o^the Kronecker product of matrices 
which ,can be used to characterize all solutions of certain 
classes of p r obj^ms^thaf' o^ccur in the design of experiments 
and in lifiear programming. 

DEFINITION 2: For a\ny m by n matrix, P, and s by t 
matrix, Q » ^lcty^' *V ^ ronec ^ er product of P and 
Q is the ms by nt matrix-, PXQi* °f * ne form 

PX9 ■ (q kl P), 
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It should' be noted in Definition 2 that if p[- (p..) 
and Q « (q ki ), then PXQ is obtained by replacing j eact/ele- 
ment fa by the matrix q^P, whereas QXP is) obtained by 
replacing each element Pij . by the matrix -ft^Q. Consequently 
PXQ and QXP differ only in the order in which r<jws and 
columns appear, and there exist permutation matrices R and S, 
say, such that QXP = *(P X QlS. (We remark also that some 
authors, for example, ThTall-WNTornheim [13], define the 
Kronecker product of P and Q alternately as P X Q = (p.-Q), 
that is, our^QXP. In view of the discussion in Section 3.1 
of the Moor/- Penrose inverses of matrices A and A, wh'ere A 
is obtained/ by permuting rows of A, columns of A or both, * 
each of the following results obtained using the form for 
PXQ in Definition 2 has a corresponding dual if the alternate 
definition is employed.) 

Example 3.4 ' v . 

If * , 



then 



pXq 



and 



QXP 





"1 


# 2' 






"0 


* 4 




1 : 




2 






> Q 












i 




/ 


0_ 






2 


i 




3 










\ 












r ♦ 










\ 














"0 


0 


4 


X B 


-1 




2* 




a 


0 


0 


12 




0 


-3 


0 






2 


4 


i 


2i 


- 3* 




5 






.6 


0 


3i 




0 


'* 9 


' \ 


i. 




"0 


4 


~-l 


0 




6 


-2 






a 


2 


i 


3 


4 




2i 


6 








0 


12 


-3 


0 




0 


0 








6 


3i 


9 


0 




0 


0 










/ 




f 











Given any Kronecker product PXQ, it follows fro 
Definition 2 that 



(3.6) 



IPXQ] 



H 



(q 



p H ) 



p h x Q ^ 



iff'-.' ' 



Also, for any matrices R and S - Ts^) with the products PR 
and QS defined*, the product^ IP X Ql IR X S| is defined, and we — 
have by use of block multiplication that 



IP XQ1 IRXS] 



q u P • • • q ln P 



•s n R 



( 



s nl R 



s lt R 



s nt R 



Therefore, ✓ . 

(3.7) IPXQ1IRXS] - PRXQS. 

The following lemma can be established simply by// combining, 
the relationships in (3.6) an4 (3.7) to show tha/t the defining 
equations in (2.2) are satisfied. 

LEMMA 7: Fo^ any matrices P and/Jft IpXq]* * P^Xq*. I 

Example 3.5 

Jxr construct if 
* 



A - 



'2 0 1 4 0 m 2 

2 .3 .-1 4-6-2 

6 , 0 3 8 0 4 

6 V 9 -3 8^2 -4 



observe that A - PXQ» where 

J I 2 0 il • fi 21 . 

k 3 4- *•[> J- 

i 4.9 

I: ERJC 



Then we have 



•1/2 



* 4 -2" 




122 


4" 






-9 


15 


_ : 3 1 




17 


-8_ 



so that 



r 


















88 


16 




-44 




-r 




















"T 


36 


60 


J 


18 




30 




. "* 


A 




P 


+ XQ + 




1 

127 




o8 
66 
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L L 


16 
4 






















11 
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- 9 




15 


V 




















51 


24 




17 




-$ 


















* 


- 
















• » Example 


? 


.6 




















a 








' \ To 


construct 


A + 


if 


















• 


<• 






1 


0 


0 


1 


0 


1 


6 


0 




0 


(f 




* 


/ 






0 


1 


0 


1 


0 


0 


1 


0 , 


0 


0 


0 






v » 

% A = 






0 


0 


J 


1 


0 




0 


, 1 




'0 


0 












1 


0 


0 


0 


1 


0 


0 


0 


i 


* 


0 












0 


1 


0 


0 


1 


Q 


0 


tf • 


0 


1 


0 












i 


0 


1 


0 


1 


0 


0 


0 


0 


0 


1 







observe first that if denotes the vector with all i 
elements each equal to unity, then A\can be written in . 
partitioned form as 



(3-8) 



A 3 

0 ' * 



Whereupon, the'fjrst two columns of A in (3.8) can Jbe 
written as the Kronecktff product tu£ I 3 1 # -X u 2 . Next observe 
that permuting columns ,"of A" to* form 



?3 



h- "3 Js 



u 3 *3 0 * 0 «3 »3J 
the last four, columns of A" become <u 3 ,I 3 ] X I 2 - flferefore - 



0*- 0 
I 
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A can be written as "* 

A » [u^I^ ] X [u 2 ,I 2 ], 

and thus 

(3. a) A + » [u 3 ,i 3 ] + X [u 2 ,i 2 ] + . 

Permuting rows of [I,u] + in Exercise 2.10 now gives 

H 



Lu.,1.] 
x 1 J 



1 

i + 1 



(i+l)I.-u.u.l 
v ' i li 




so that (3^3) becomes 



) 



(3.10) 



1 









\ < 1 




»2 H 




X 




4I 3- U 3 U 3 H 




3I 2" U 2 U 2 H 



Substituting numerical values into (3.10) 
mutation of rows of A + yields A + . 



a.suitable per- 



Matrices,- A, as in Example 3.6, with elements 2ero or 
one occur frequently in the statistical design of experiments , 
and the^techniq,ue of" introducing Kroneirker products can often 
be used .to construct A + , and thus all solutions of systems of 
equations Ax^~~J) by use of Exercise 2.21. The additional 
restrictions on ^x=^b -^o^obtain solutions with particular 
properties can then be formula! 

N(A) or, equivalent ly, I-A + A. *(See ExerciseN^. 11 . ) 

That Kronecker products can be combined wijth forms^for 
Moore-Penrose inverses of partitioned matrices/to construct 
A for other classes »f \triirtn T f>H TiTitri rr-> i i ilmun by the 
representation in Theorem 8. 



terms of conditions on 



THEOREM 8: 0 Let W be any m by n matrix, and for any positive 
integer p let = (pI n +W H W) " X . Then 



(3.11) 



I Xu n 
n A p. 



wx i 



p J 



I6 p x V w > p ■ g p w+ XVp" ] - ' 



^oo/: Observe first that with 

( P I n +W H WHI n -W + W) =.p(I n -W + W) = (i n -w + w)( P i n+ w"w) 



then 

:(3.12) , V wV- P G p (I n -W + W lss PP n -W + W)G p , 
Also, observe that with CpI n +W H W)W + = P W + ♦ W H , the relation 

(3.13) W + » pG W + + G W H , * 
P P * 

together with the fact that G p is Hermitian, HUnpiies 



. P 

is Hermitian 
Let 



n A p 

w Xi 



and let 



Then it follows from (3. 12) , and (3.7) that 



= g p x Vp H + w+w Xi p -g p w +w XVp H 

= Gp (I n"^Xu p u p H + W + wXl 



' F(V w+ ^Xu p u p H + w + wxi f 

is Hermitian. Also, with u H u - p, 

P P 



i ERIC 
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AXA 



£d n -W + W) Xpu p H * W + HXu p H 



iw(I -W + W) X"'u " + WW + WXI„ 
p v n PP ,v p 





"i p Xu p «" 















Continuing, we have 



v 



^(G.XuJ - W W)G p Xpu p +W WG p .Xu p - G p Xu p , 



p " p' p^ n 



P v n 



P P 



and 0 

XA(G/Xu p u p H ) - -J-(I n -W + W)G p W + Xpu p u p H + W*WG p W + Xu- p u p H 

- GW + Xu p u p H .^ 
Hence, XAX ■ X. Finally, farming AX gives 



AX 



W + .X u p H - G p W + X pu p H 



L WG p Xu p ww + Xi p - wcrw + Xu p u p h 



Now 



W + Xu p H - G p W + X,pu p H - G/Xu p . H , 



by (3.13), which, with G p and WG p W + Hermitian, implies • 
(AX) H « AX. 

Having shbwn tha£ A and X satisfy the equations in (2.2) ^ 
then X * A + which establishes (3.11). I 

Example f 3.7 

If p » 3, then for any m by n matrix, 





m 


\ 

K 


! n 
0 


V 

0 


WXI 3 




0 
,0 


. w 

0 


0 

w 
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•is 



and 



wXr- 



G 3 






" G 3 W+ 


S 




W*-G 3 W* 


-c 3 w* 






-G 3 W + - 


W*-G 3 W + 


-1 









Suppose" now that we let T - I(p,W) denote the matrix in 
Theorem 8 which is completely determined by p and the sub- 
matrix W, that is, ) 



T ■ T(p,w) - 



n " p 



Now given a syste3 of equations Jx « b with W m by n of**rank 
r, 0 < r < n f and p any positive *iaiteger it follows that 'if 



we partition x 


and b 


as 










x * 




, b - 










> Crt . 



,(1) 



(p) and b (0) n-tuples and 



with x v *' f . . . f x 
then x is a solution if and only if 

(3.14) £ - b(°) 
j-1 

and . 



.'(3.15) 



Wx 



(J) . kO) 



j • 1, 



,p. 



\ 



. ,b^ m-tuples 



other words, each x^ must be a solution of m equations" 
n unknowns, subject to the condition that the sum of the 

J Th^se characterizations are 



solutions is equal to b ! 
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further explored for the general case of an arbitrary matrix 
W, in^Exercises 3.16 and 3.17 and for an important special 
case in Exercises 3.18 and 3.19. 



Exerci ses 

3-10 Complete the numerical dons t ruction of A + in Example 3.6 and 
* verify that A and A satisfy the defining equations in (2.2). 

3.H Matrices of the form 4u p ,I p ] and, more general ly, Kronecker 
£ products such "as- A in Example 3.6 in which at least one of the 
matrices has this form occur frequently in statistical design of 
experiments [I] [A]. For example, suppose it is required to 
examine the effect of p different fertilizers on soy bean yield. 
One approach to this problep/is to divide a field into ptj subsec- 
tions (catted pi ots) , randomly assign each of the p type of 
fertilizers to q plots, and measure the yield from.each. Neglect- 
ing other factors which may effect yiejd, a model for this 
experiment has the form , 

(3-16) « y. . * m + -t. + e. . | 
ij 1 ij 

. where y_ is'the yield of the j th plot to which fertilizer i has 

been applied, m is an estimate of an overall "main" effect, t. i£. 

an estimate of the effect of- the .part icular fertilizer treatment*^ 

and e.j is the experimental error associated with the particular 

— "HPlot. * The -question now is to determined and t , t to mini- 

I P 

mize the sum of squares of experimental error, that is, 

" ! - • 

j fm\ j-1 ,J 



If y and e denote .the vectors 



(y 1l V y l2""* y p2 y lq'--" y pq ) 



H 



and' 



6=(e |l e pl' e l2 , -'--' e p2 , --; ,e lq , ""' e pq )H ' 
show that data for the model in (3. 16) can be represented as 
(3.17) y - Ax + e, 
1 where x - '(m,t, t p ) H and A - [u p ,I p J Xu q . 

o - - 48 - 
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1' 

b, SHow that r(A) ■ p, an4. construct the minimal norm solution 
* - A + y, to (3.17). 

c. « For % statist ical appHcat ions it i s al so assumed that 

e 

p 

J t. -0. ' 
•I I 

Starting with x from V*41b^afep_Ye , construct that solution 
say, for which this additional cond i t ion holds . 

3.12 (Continuation): Given the experimental situation described in 

Exercise 3.H, it is sometimes assumed that there is another * 
effect, celled a block effect, present. 1n this case one of two 
models Is assumed; First, if there is no Interaction between the 
treatment and block effect, then . » * 

(3.18) y*i M m + t. + b. + e ,. t 

whereas If there is an assumed interaction between the treatme^ 
and block effect, then- 



(3.19) 



V,j - t. ♦ b. + (tb).. + e.., 




where y^, m and t. have the**saroe meaging as in (3.16), 
b j tj"l » . . . ,q» designate block effects, and ( tb) . . , f«| , . t p and 
j«l,...,q, designate the effect of the interaction between treat- 
ment j and block j . 

a. Using the notation for y and e from Exercise 3.11, show that 
the data for the model in (3.18) can be represented a$ 

y ■ AjX + e, » J 

where now x - (m,.t. ,...,t ,b.,.,.,b ) and 

I p I q ^ 

^ i l p A q p A qj 

b. jShow that the data for the model in (3.19) can be represented 



y » A 2 * + b ' 

where - , • -m 

X " (m ; t | Vl b q" (tb) n (tb) ..q t^Pl"- % - (t , b) pq )H 
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V 

-^-[i%.VX V %xi q .i,xiJ. ^ 

c. Use the procedure of Example 3.6 to construct A 2 + and thus 
the solution x » A 2 + y. (For statistical applications the 
mode! in (3.19) is not meaningful un+ess there is more than 
one observation for each pair of indices i and j, that is, a 
model of the form 

y ijk -«m V . + b. + (tb),. + e. jk 

where k - t,:..,r. In this case the unique solution Is obtained 
by assuming ^ 

• I"tj- J b - 0 and also f Ctb) . . - L(tb)..*0 



for all i and j. Note, in addition, that the construction of 
Aj in 3- 12a above is somewhat more complicated but can be 
formed using^related techniques. The particular solution used 
for statistical applications in this case assumes that 

' I t- - 'I b - 0.) 
* • i-l j-1 J 

*3.13 Show that if P and Q are any square matrices with x an eigenvector 
of P corresponding to eigenvalue A and y an eigenvector of Q cor- 
responding toiigenvalue u, then x\y Is an eigenvector of P\Q 
correspond! rf§ to eigenvalue Ay. «• 

3.H a. Show that for any p and W, I-T*T - (I Vv) X'O -u l u + ) 

n P P p 

*b. Construct a complete orthonormal set of eigenvectors for 1-T*T. 

3.15 Prove directly that for any mbyn matrix W of rank r.and any p, 
rank'fl^- n + r(p-l). 

3.16 For any system of equations Tx - b'wlth x and b partitioned to 
give (3.14) and (3.15), let X and B denote the, matrices 

x- bSV : . 

a. Prove that Tx - b if and only If there exists a matrix X such 
that * 4 
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f 



(3.20) Xu - h M 

P 6 

(3.21) WX - B. 

b. Prove that a necessary condition for a solution, X, to (3.20) 

and 13. ZD to exist is WW + B - B and^b^ 0 ^- Bu . 

P 

*3 - 1 7 (Continuation): For any eigenvector ZjXyjOf I^'T^Tt where 4 
Yj has components , . . » tyj p T let Z.j denote the matrix 

z ij -[y ji z i y j P r i ] - 

Show that z. X y j corresponds to eigenvalue XjYj «■ 1 If and only if 
UP 1 

anjl * 

(3.23) WZ.j - 0. b /J 

3.18 The transportation problem in linear programming is an Example of 
a problem In which it is required to solve a system of aquation* 
Tx * b- This famous problem can be stated as follows: Consider 
a company with n plants which produce dj,...,a n units, respectively, 
* of a given product In some time period. This company has p 

• distributors which require b units, respectively , of the 

product In the same time period, where 

s 

^ i-i 1 j-i J 

If there is a unit cost c j j for shipping from plant 1 to distributor 

j, l»1,...,n and' j«1 p, then how should the shipments be alio- 

cated In order to ml n tmi 2e ^otal transportation cost? This problem 
can be illustrated in a schematic form (called a tableau) as shown 
In Figure 6 where 0j,...,0 n designate origins of shipment (plants), 
* - 0^,..., 0 designate destinations (distributors) and for each 1 an J 
j , *X|j denotes the number of units to^be shipped from 0j to Dj . 

The problem now is to determine the x^j, i-1,...,n and j a 1,.~.,p 
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to m1niml2e the total shipping cost. 
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'il 




> 






ip 




• 




' ' ' 




. • . 




• 


0 

n 


C nl 






c . 
nj 






c 

np 






> 






nj 




X 

np 


a 

n 












b 

P 


h • h 

■ J J 



Figure 61 The transportation problem tableau. 



(3-2M I Ic.k... 

1-1 ji| J 'J 



subject to the condition^ that 



(3.2S) £ x, . - a., I - l,...,n, 



and 



n 



(3-26) [ x . - b , j. - I p. 

i-l u J 

Also, we must have x^'j ^ G, for all 1 and j f and, assuming frafc- 
tlonal units cannot be' manufactured or shipped, all a } , bj and x { j 
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^Integers. (This last requirement that tne- x. . are integers 
follows automatically when the aj and bj are' 16].) 

Show that If the x.^ In Figure 6 are elements of an n by p 
matrix X, and*!/ ^o). [a, , a * ] H , then the conditions in 
(3.25) can be written as Xu p ■ b;°*and the conditions in*(3.26) 
become 

" n H x- [b,,...,b p ]. , 

Therefore,' (3.25) end (3.26) together imply that any set of 
numbers x^j which satisfy the row and column requirements of 
the- tableau is a solution of Jx • b where T . T(p, u fl ) an( j 

b * £a i';--' a n' b i V H - 

} b. Prove that If T - T(p,u H ), then 

f 1 / 



Moreover, show that lf~x. . Is the element in row i and column 
j of the tableau form of x* ■ then 



n 



x. . ■ —a. + 



lj .p - l " n D j " n? 



a. 



1-1 



i 



,n and j«1 d« 

*c. Show that rank (|) ■ n + p- 1 when v/ - u n H , and thus rank 

{I np " FT> * <n-D(p-1). Also, construct a complete otrho-/ 
normal -set of eigenvectors of I ^ - J ¥ J J and show that Zj/jjfy. 
Is an "eigenvector corresponding to eigenvalue A.y. - 1 if and 
^ only If all row sums and column sums in the tableau fopm are 
zero. - * t 

d. The, vector g - (* np - J*7)c is called the gradient of the 
inner product 

' (C,X) - I 'l Ce.X.e 

,1-1 J-1 ' J ' J 

I 

In (3.2A)t. Show that the elements , l,n the tableajjtorm 
for g can be written as 
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1 ? .if 



for i » l,...,n and j*l,...,p. 



c. . + — 
Ij np . 



I k 



3-19 (Continuation): The transportation problem has been generalized 

in a number of different ways, and one of these extensions follows 
*■ directly using matrices of the form J * T(p»W) . Suppose^ that we 
are given q transportation problems, each with n origins\and p 

destinations, and let a7k ,b jk' C ijk a " d X ijk be the row S T S ' co1 " 
^ umn sums, costs and variables, respectively, associated with the 
k tableau, k»l,...,q. A "three-dimensional' 1 transportation 
problem is now obtained by adding the conditions that 



k-1 



.jk M ij 



,p, where d. 



for i»l , . . . ,n and j*l , . 
integers. 

apparent by noting that if the tableaus are stacked to form a\ 
paral lelopiped with q layers each with np cells, then (3.27) 
simply implies, np conditions that must be satisfied when the 
are summed in the vertical direction as shown In Figure 7, wheVe 
only the row, column and vertical sum requirements are indicated.) 

a. Show that the conditions 



j||,...,d are given positive 
(TheTtroice of nomenclature "three-dimensional is\ 



Vjk 



* n p 

I a ik - I 
i-i ,k j>i 



Mr 



k-J 



k-1 



'jk 



n 

i-i 



J-i, 



.n, 



are necessary in order for a three-dimensional transportation 
problems to have a solution. 

Show that the conditions which the must satjsfy If there 

is -a solution can be written as T* " b where J * T(<J*W) with 
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Figure 7. The paral lelopiped requirements for the 
tableau of a three-dimensional transportation problem. 



w * T^j ■ T(p>u n H ) the matrix for the "two-dimensional trans- 
portation problem in Exercise 3.18 and a sui ta$l£ vector b. 
Show that 



q P np »np 'npj 

' J'p -'iVp^'n 








n . 


y 



*nd that G q T np + - fU,V] where 



55- 



ii - J It 2n ft p t q h v 

U ' nTn+oJ^p Tp+q)(n+p+q) U P U P J X" n 

M2 



-»v). 



and 



V . Ij V J J I - n + 2p + q H| 
rp A n(|>+q)i n (n+p) (n+p+q) u n u n J' 



3.3 Miscellaneous! Exercises * * 
' \ * i 

3.20 Prove that a necessary and sufficient condition that the equations ^ 
AX ■ C, XB ■ 0 have a common solution is that each equation has -a 
solution and than AO • CB, in which case X - A + C + DB* - A + A0B is 

4 a particular solution. 

3.21 Prove Lemma 7. 

3.22 Prove .that 

I |li B H B - B+b] il + B+b"*] - I + B H B 



I 

> 



for any matrix B, andl that 



J (f + B H B )-' + (i)b + B H+ )-' -21 - B + B. 



9 
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4 

Drazin Inverses 



1 T* V 



4.1 The Drazin fhverse of a Square Matrix ■ ■ 

. < ~~ \ \ 7] 

In this sec^on we^ consider another type of . generalized 
inverse for square complex matrices. hne inverse in Theorem 
9, djje to Drazin [31, has a^variety of\ applications 



THEOREM 9:, For any square matrix, A, there is a uhique ' / 
t matrix X such rJvat \ ' ' *j 

(4.1) A - A X, for some positive integer k, K 
■ (4.2) xV- l % / ' / ' / ^ r , # 

(4. -3) AX - Jfl. * ' ^ 

Proo/; Observe ^irst that if A = 0 is die null .matrix, then 
A and X ■ 0 'satisfy (4.1), (4."*) and (4.3).* * 

Suppose A j** 0 is any hbyn matrix'. Then there exist 
scalars,^,,.; . ,d t , not all zero, such that - * 

■* , " • . -57- * , 

r ERJC -V - \ < 



2 

\, where t < n +1 since the A 1 can be viewed as sectors with n 
elements. Let be the first nonzero "coefficient. Then we 



can write 

(4.4) 
where 
# 



A k » A k+1 U, 



t 

I d.A 
iSk*l 1 



i-k-l 



Since U is a 'polynomial in A*, U and A commute. Also, multi- 
plying both sides of (4.4) by AU gives 



A k » A k+2 U 2 



aK*V 



and xthus 



k 



(4.5) . A" » 
; for all m £ 1. 



.k+m-.m 
A u 



and 



Let X.= A k U k+1 . Then for, tttis choice of X, 

A k+1 Y A 2k+l„k+l A k 
A X = A u ■ A 



X 2 A » A k U k+1 A k U k + 1 A = (A 2k+1 U k + 1 )U k+1 = S*^ 1 - Jf,' 



by use of (4.5). Also, X and A commute since U and A commute. 
Thus the'Qonditions (4.1), (4.2) and (4:3) hold for this X. 



To show that X is Wique, suppose that Y is also a 
solution to (4.1), (4.2) and (4.5), where X corresponds to 
an exponent k 1 and Y corresponds to an exponent k 2 in (4.1). 
Let k = maximum (k 1 ,k^). -Then it follows using (4 . 1) , 1 (4. 2) , 
(4.3) and (4.50 "that * 



v k y k + l * 



i - X 2 A - X 3 A 2 - ..l- X k+1 A k - X k+1 A k+1 Y 

- XAY - . . . 
' * ... - AY 2 ■ Y 2 A - Y 
to establish uniqueness. | 
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We will call the unique matrix X in Theorem 9 tfoe Dvazin 
inverse of A and write X alternately as X » A* d . Also, we m 
will call the smallest k such that (4.1) holds the index of 
A. ' , 

Tha.t A d is a generalized inverse of A is apparent by 
noting that (4.1) holds with k « 1 when X - A -1 exists and 
also (4.2) and ("4.}) hold. Observe, moreover^ that in 
general (4.1) can be rewritten as' 

(4.6) A k XA « A k * 

and (4.2) becomes XAX » X, by use of (4.3), so that the' 
defining equations in Theorem 9 can be viewed as an alterna- 
tive to those used for A + in which AXA = A is replaced by 
,( 4 - 6 )» (1.2) remains unchanged, and (1.3) and (TTV) are 
replaced by the condition in (4.3) that^A and X commute. 
(Various relationships between A d and A + will be explored 
in the exercises at the end of this section and in Section 

As will be discussed following^ the proof of Lemma 10, 
full rank factorizations of A can be used effectively in 
the construction of A d . * 

% ^ 

LEMMA 10; For any factorization A » BC, A d = B(CB) d 2 C. 

Proof: Observe first that for any square matrix A and posi- 
tive integers k,'m and' n, we have A, m A n = A, m " n if m > n and 
A A d «A if m > k and A has index k. 

Let k denote the larger of the index of BC and the 
index of CB.^ Then 

. A d ■ < BC ) d " CBC^CBC)/* 2 = B(CB) k CCBC), k+2 

- B(CB) d k+2 (CB) Zk+2 C(BC) d k+2 

B(CB) d k+2 C(BC) 2k+2 (BC) d k+2 = B(CB) d k+2 C(BC) k 
p . * . » 

- B(CB) d k+2 (CB) k C - B(GB) d 2 C. | 

' « -S9- 



Suppose now that A - is a full rank factorization 

where rank (A) « r 1 . Forming the r 1 byr 1 matrix C 1 B 1 , then 
either C 1 % 1 is nonsingular, or CjB^- 0, or rank (C^) » r 2 
where 0 < r 2 < r^. In the first ca^e, with C 1 B 1 nonsingular, 
(C 1 B 1 ) d * (C 1 B 1 )" 1 so that A d » B x (C 1 B 1 ) ' 2 C X , by Lemma 10, 
where 




On the other hand, if « 0 then (C 1 B 1 ) d - 0 and thus x 

A d a 0 by again using Lemma 10. Finally, 4f rank (CjBj) 
= r 2 ,0 < r 2 < r Jf then for any full rank factorizati; 
C 1 B 1 ■ B 2 C 2 , we have 

tWdJ B 2( C 2 B 2^>d 2c 2 

so that A d in Lemma lO^becomes A^ ■ B 1 B 2 (C 2 B 2 ) < j 3 C 2 C 1 . 'The" - ^ 
same argument now applies to C2B 2 , that is, either C 2 B 2 is 
nojvsingular and • % ) 

(C 2 B 2 ) d 3 (C 2 B 2 3* 3 , 

or C 2 B 2 = 0 anil thus A d - 0, or rank (C^) » r 3 where 

0 < r 3 < r 2 , and = BjCj is*a full rank factorization to 

wh^ch* Lemma 10 can be applied. Continuing in this manner 

with 

» p m 4 

ran* (B^) > rank (C i B i ) * rank (B i+1 C i + 1 )> i«l,2,..., 

v * 

✓then either B C * 0 for some index m, and so Aj * 0, or 
mm Q 

rank (B C ) * rank (C R ) > 0 for some index m, in which case 
v m m' v m nr 

(B C ) , - B (C B ) " 2 C 

v in Vd m v m nr m 

and thus 

(4.7) A d - M 2 ... \(W m -\c m _ t ... Cj 



A 2 - BjC^Cj - B^C^ A m - ,.. B^.j ... Cj 



in Lemjna 10. Observe, moreover, that with A * B i c i» 
A 2 - 
and 

r 
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(4.8).. 



*m+l 



B l»2 



m v m nr m m-1 



... C, 



we have either A m = A m+1 » 0, ancl A d » 0, or that A d has the 
form in (4.7) where, since each has full column rank and 
each C i has full row rank, * 



and 



Vl 



B C 

m m 



... b/a^V 



= C B . 

m m 



Therefore, in both, cases we have rank (A m ) » rank (A m+1 )« 
Further mo re, Tilfo^^wT'iiTbotn cases tha* (4. 1) 'holds Tor- 



k « m arid does not hold for any k < m. That is to say } k ir 
(4.1) is the smallest positive integer jsuch that A k and A k ; 
have the same fank. 



Example 4.1 

If A is the singular matrix 



written as the full rank factorization 



A - B lCl r 



-2 
1 



then 



is nonsingular, so that A has index one, and 



B 1 (C i B 1 )- 2 C 1 * £ 



"6 


•26 


30" 


3 


35 


-33 


3 


3 


-1 



E xample 4.2 

If A is the matrix 



0 0,0 
0 1 ,2 
0 0 ' 3 



0 0 0 



1 



with A = BjC 1 the'full ra'nk f actorizatioa, 



A = BjCj 



where 



C 1 B 1 



7 0 0 

0 1 0 

0 1 1 

0 0 0 



7 , f 0 0 
0 1 1 
0 -1 -1 



"10 0 0 
0 0 12 
0 0-11 



then rank (CjB^) ^ 2 and 



C 1 B 1 ' B 2<?2 



7 0 
0 1 
0 -1 



10 0 
0 1 1 



is a full ,rank factorization. Continuing, 
so that 



C 2 B 2 ft 



C 2 B 2 » B3C3 



[1 0] 



is a full rank factorization with CjBj ■ 7. Hence A has 
index three and A^ becomes 
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A d -B 1 B 2 B3(C 3 B3)- 4 C 3 C 2 q i 

[i q >o o] 



1 

ZTUT 



343 
o 
o 
o 



y 4 * 0 0 0 

o "o 0 0 

0"^0 0 0 

0 0 0 0 



For the special case of matrices with indox one we have 



(4,9) 

so that 
(4.10) 



AA d A s> 



A, 



AA. = A, A, 



CA d ) d - X 



by the duality in the roles of % *A and A d . Conversely, if * 
(4.10) holds, then, the first and last relations in**(4.9) 
folio* from the defining relations in (4.2) and (4.3) -applied 
to ( A d^d and A d' and the second 

relation in (4.9} is simp ly 
(4.2) for A d and A. Consequently, (4 . lO^holds if and only 
if A has index one. In this special case the Dra2in inverse 
of A is frequently called the group inverse of A, and is 
designated alternately as A# . Thus X * A# , when it exists, 
is the unique solution of AXA ■ A, XAX - A and AX « XA, and 
it follows from Lemma 10 that for any. full rank factorization 
A « BC, A # » B(CB)"2c. 



Exercises 



Compute A d for the matrices 





"7 


8 


5 




a 2 3* 


A j *• 


k 


5- 


3 


■ A 2 - 


0 0-1 




5 


7 






0 0 0^ 



V 



I 12 
10 1 
0 I 0 
-2 -2 -3 



Given any 1 matrices B and t of the same size where B has full 
column rank, we will say that C'ls alias to B if B + * (C H B) + C H .» 

a. Prove that If C Is alias to ft, then C H B Is nonslngular. 

b; w Show that the set of all matrices alias to E? fo>m an 
equivalence class. 



c. Prove that A . - A if and only if C is alias to B for any 
ful! rank factorization A ■ BC . ' 

v 

d. Note, in particular, that A d « A + when A is Hermitian. 
Prove this fact directly and also^y us*ing the result in 
4.2c above. . ^ 

H 



k.3 Prove that (A H ) d » A d " and that 



(^ A d'd)d * A d for any matrlx A « 

Prove that A rf = 0 for any nilpotent matrix A. :« 

k\S Prove that [PXQJ d - P d XQ d for any square matrices P and Q. What 
is the i ndex of V X Q? *» ' 



4.2 An Extension to Rectangular Matrices 

c 

, The Drazin inverse of a matrix, A, as defined in 

Theorem 9, exists only if A is square, and an obvious question 
is how this^ defiiyitirfh can be extended to.- rectangular matrices.. 
One approach to this problem is to observe that if B is a 
mbyn i&fch m > n, say, then B can be augmented by m-n columns 
ot zeroes to form a square matrix A, Now forming A d , we 
might then take those columns of A d which correspond to the 
locations of columns of B in A as a definition of the "Drazin 
inverse" of B. As shown in the f o'llowing^example , however, 
the difficulty in this approach is that there are m such 
matrices A, obtained by considering all possible ^ m " nj 
arrangements of the n columns of B (taken without any ^ermuta- 
tioas) and the m-n columns of zeroes, and that A d can be 
different in each case. 

Example 4.3 

If 

' J 1 
B = 



/ 



and 



A » 
A l 



"0 


1 


2 




"1 


0 


2 




"1 


2 
















V, 










0 


0 


1 


, A 2 » 


0 


^0 


1 


• A 3* 


0. 


1 


'o 


0 


3 


•1 




3 


0 


-1 # 




3 


-1 






















s 





0 
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then 



CAW) 





'0 


10 


7" 






_ 1 


0 


2" 


1 


0 


3 


3- 




1 

-7 


0 


0 


1 




_0 


9 


0 




3 


0 






"1 


-2 


0" 














0 


1 


0 














3 




0 













are obtained by applying Lemma 10 to the matrices A. 
where t . 



BC 



1 

0' 



C 3 * 



Observe in Example 4.3 that the nonzero columns of each 
matrix (A i ) d correspond to the product B(C i B) d 2 . Conse- 
quently, using the nonzero columns of (A^ to define the 
M Drazin inverse" of B implies that the resulting matrix is a 
function of C i . That such matrices are uniquely determined 
by a. set of defining equations and are special cases of a 
class of generalized^inverses that can be constructed for* 
any matrix B will be ap^axtfnt from Theorem 11. 

f THEOREM 11: For any m by h matrix B and any nbym matrix W, 
there is a unique matrix X such that 

(4.11V (BW) k * (BW) k+1 Xw\ for some positive integer**, - 

(4.12) XWBWX » X, ' 

(4.13) BWX » XWB. 

Proof: Let X - B(WB) d 2 . Then with XW - *B(WB) d 2 W = (BW) d , 
by Lemma 10, (4.11) holds with k ^he index of BW. Also, 

XWBWX » B(WB) d 2 WBWB(WB) d 2 , » B{WB) d 2 - X. 



£WX • BWB(WB) d 2 * B 




so that (4*12) an^ 
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To- show that X is* Unique we can proceed as in the proof 
of Theorem 9. Thus, suppose X 1 and X 2 are solutions of 
(4,11), <4 .12) and (4,13) corresponding to positive integers 

and k 2 , respectively, in (4.11), Then with 
k = maximum (k^,k 2 ), it follows that 

X x = X 1 WBWJC 1 = BWX 1 WX 1 =.(B^') 2 (X 1 W) 2 X 1 

= tBW) k tX 1 W) k X 1 = (BW) k+1 X 2 W(X 1 W) k X 1 

WB) k+ ^W(X 1 W) k X 1 - X 2 WBy(BW) k (X 1 W) k X 1 



x.hbwx 



^Continuing in\ a similar manner with 
X 2 (WX " k+3 



\ 




= X^(WX 2 ) k+1 W(BW) k B 
= X z (\WX2; k+1 (WB) k+1 



Therefore, with Xj * X 2 , the solution to (4,11), (4.12) and 
(4^3) is uriique, I » ^* 

The unique matrix, X, in Theorem 11 will be called the 
W-welgJited Draain invevve of J3 and will be written alternately 
as X\= (B w ) d . , ; 

he choice of nomenclature W-we?fehted Dra^in inverse of 
B is kasily seen by noting that with (B w ) d - B(WB) d 2 , then 

wl ■ 

73 < 
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(B w ) d * B d when B is s<juare and W is the^iwHrtty matrix. 
Also, observe- more generally tha^v^^Band (B w ) d of the 
same size and with W andjmrthe size of B H , the relation 
BW(B w ) d = (By^WB--iir(4.13) can be viewed as a generalized 
commut^mTfy^ and (B w ) d WBW(B w ) d 

analogous to (4.2) when written in .the form XAX = X 
Example 4.4 ' " * 



(B w ) d in (4.12) 



If 



B = 



1 2 
0 1 
3 -1 



is the matrix in Example 4.3, and 



then 



4 
-2 



w 2 = 



3 
0 



.(91)' 



169 538 
60 169 
87 -169 



<Vd 



1 



Exerci ses 
4.6 



4.7 

4.8 
4.9 



Verify that^B and (B w > d satisfy the defining equations in Theorem 
II for W - C 2 , in Example 4.3 and for W * W ]4 W 2 in 
Example 4.4. \ 

\ 

Prov»e that E » for any itiempotent matrix E, and thus that 
tj i s square and W ■ B . 



(Consequently, 



(B w )^ .>p B"B d when 
( B w ) d ■ B when W L and B has index one.) » 

Show that if W H is apyVmatrix alias to B, then (B w > d - W + (WB)~ l . 

' / 

Psove that (B y ) d 



H+ + H+ h 
B B B for any matrix B when W «• B . (Note 



that this result follows at once from Lemma 5(f) and Exercise 
4.8 if B has full cdtumn rank, whereas Lemma 5(0 and Exercised 
4. 2d can be used for the general case.) 



9 

ERIC 



-67- 



74 



4.3 Expressions Relating and A ♦ 

It is an immediate consequence of Exercise 4.9 that if 
W » B H and so # 

CVd" b h Vb h+ , 

then \ 

* B + - W(B w ) d V. , I 

Thus, using/lT-weighted Drazin inverses with W ~ B H ,(B w ) d and 
B + are related directly in terms of products of matrices 
which implies that (B^) d and B + have the same rank In con- 
trast, for any square matrix,. A, we have 

rank (A d ) = rank (A k ) = rank (A k+1 ) , 

with k the index of A, whereas rank (A + ) » rank (A). There- 
fore, rank (A d ) <^ rank (A ) with equality holding if and 
only if A has a greup inverse. The following result can be 
used to give a general expression for the Drazin inverse of 
a matrix, A, in terms of powers of A and a Mtfore-Penrose 
inverse. 

THEOREM 12: For any square matrix A with index k, 1 . 4 

(4.14*) A d » A k YA k 

for any matrix Y such that 

(4.15) A 2k+ V k+1 -.= A 2k+1 . 

Proof: Starting with the right-hand* side of* (4.14) we have 

* A d k+1 A 2kil YA 2k+1 A a k+1 

A d k+1 A 2k+1 A d k+1 = A d 2k+2 A 2k * r - A d . I 

Observe* in (4,15) that oA^ obvious choice of V is ? 
(A 2k+1 ) + , and it then follows that A*,(A*) + ajv* A d ha'v? "the 
same rank for every positive integer ^> k. In $fiVs case, 
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various Relationships among A £ ,(A £ 0 + and A d can be 
. ^ established. For example, it can be, shown that for any I > k, 
there is a unique matrix X ♦tisfying ^ 

(4.16) A £ XA l » A*, XA*X - X • ' ' 
and 

* (4.17) (XA £ ) H » XA £ ? A £ * » AA d . 

Dually, there is a Unique matrix X*satisfying (4.16) and 

(4.18) (A £ X) H » A £ X, XA £ » AA d . v 

The unique solutions of ^(4.16) and (4.17) and of (4^.16) and 
(4.18) are Called the^eft and right power inverses of A £ , 
respectively, ana>*re designated as (A £ ) L and (A £ )^. More- 
over,^ it can Ire shown (Exercise 4.11) that 

.(4.19)' (A £ ) L « (A £ ) t+ AA d , ,( A \ = A d A(A £ ) + , 

and- (Exerc&e 4.J4) that (A £ ) L and (A*) R <can be computed 
using full rank factor izatiorfs. 



* Exercises 

kUo SHb**that If A and W sa.tlsfy aV - A £ and (WA £ ) H - WA £ for any. 



\ l - (A £ )V 
the dual form of this result for^A^W? 



.Positive Integef then WA* - (A*)V, and copversely, What is 
/ the 



.11 Prove that (A £ ) L ln;(4.19) Is the unique solution to (4. 16). and 
(<U7). * 

k.\2 Prove" that for every i > M, (A £ ) + *Va £ ), A £ (A £ ) d and 
" A d -(/)A'(AV ■ ' * 

4.13 Use • sequence\pf full rank factorizations 

A " B 1 C 1'"\ a2 " B 1 B 2 C 2 C 1 1 

to show that A 4 (A 4 ) + -■A k j(A k )* and (A 4 )V - (A k ) + A k for all t>k. 
t.\k (Continuation): Show that .* r 
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•<aV 



k 

n c, 



k+l-l 



k 

n c 



k+M 



an/ 





' k ; 




' k ' 




n b. 




>n b. 




1-1 1 




i-i j 



4.15 Construct (A 2 ) L and (A 2 ) R * fpr the matrix A in Example *.\ 

k. 16 -Prove that if Ax ■ B'is a. consistent system of equations .^nd If A 

has index, one, then the general solution of A n x ■ b,'n - 1,2 

r can be written as x - A R n b + (I-A^A)y where y is arbitrary! (Mote 
that this expression reduces to x « X" n b when A Is- nonsing^lar. 
The terminology "power inversV' of A was chosen since we use 

P ° W ^ S *°* *R ^ n a s,m ^ ,ar to obtain a particular solution 

qf A^X- b.) * 



4,4 Miscellaneous Exercises 

4.17 Let B and W be any matrices, m by n and rt by m, respectl^ 
let p be any positive integer. 

st there Is'a unique matrix X such that 



(BW) d XVT- (BW) M BWX - XWB, BW(BW) d X - X, 
a untque^-matrix X suchVthat 

XW - BW(BV) d P , WX - ,WB(VO) d P , XW(BW) P " ? X - X, 

and^hat the unique X which satisfies both sets of equations 

Is X - B(WB) P . 

d 

t b. Show that If p > 1 , q > ♦ ! and r j> 0 are Integers such that 
q> 2r + 2 « p t and If (VB)** -~(WB> d when q,« -1, then • 

- f B(WB) d P f- p(WB) q C((WB)^W)(B(WB) q )] d 2 . 

*• (Consequently, the unique X *ln 4. !7a Is the, (WB) r W-welghted 

Dra2ln inverse of B(WB) q .) % J 

\ 1 

4.18 Prove that If A and B are any matrices such that A d 2 - B^ 2 , then 




AA d - BB d . 



or Generalized Inverses 



5.1 Inverses That Are Not Unique 

Given matrices A and X, 'subset^of^ the relations' in 
(1.1) ,to (1.5) othej than those used to define A + and A d 
provide additional types of generalized inverses. Although 
not unique, some of tfiese generalized inverses exhibit tHe 
essential properties of A + required in various applications 
For example, observe that only the coiAition AXA « A was * 
needed to characterize consistent systems of equations 
Ax « b by the relation^ AXb « b in (2.4). Moreover, if A 
and tfalso satisfy frA) H - XA, then XA - A + A, by Exercise 




4.10, and with A b a "particular "solution of Ax « b, the 
general solutibn in Exercise 2.21 can be written as' 



A + b 



with the orthogo 
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♦ (r-XA)y 
decomposition 
2 



||A + b|| 2 + ||(I-XA)y|| : 
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(Note that "this is an extension of the special case, of 
matrices' with fu^prow rank used in the proof of Theorem 2.) 
In this section we consider relationships among certain of 
these generalized inverses in terms of^full rank factoriza- 
tions, and illustrate the construction of such inverses with 
numerical examples. 

For any A and X such that AXA = A, rank (X) > rank (A), 
whereas XAX = X implies ranjc (X) _< rank (A). The r f ollowing 
lemma characterizes solutions of AXA ■ A and XAX = X in 
terms of group inverses. ' 

-\ * 

L£fclMA 13: For any £ull rank factorizations A = BC and 

X = YZ, AXA * A and XAX = X if and only' 1 if AX =* (BZ) # BZ and 

XA = (YC)'YC. 

Proof: If ^A f BC and X * YZ are full rank factorizations 
where B is mbyr,,C is rbyn, Y is n by s and Z is sbym, then 
AXA * A implies * 

• (s.i) • cyzb * i ; 

and XAX ■ X implies 

(5.2) ZBCY = I s . $ « rj 9 m 

Consequently, with r * s , ZB = (CY)" 1 so that- 



and * 



AX = BCYZ - B(ZB) -1 Z - (BZ) # BZ 



lA » YZBC » Y(CY)" 1 C^- (YC) # YC, 



by Lemma 10. » 

Conversely, since Z and C have full r6w rank, (BZ) # BZB*B 
and (YC) # YCY ■ Y. t Hence AX - (BZ) # BZ gives^ AXA ■ A, and 
XA = (YC/'yC gives XAX - X. | 
v 

* Jt should be noted that the relation in (5.1) is both * 
necessary and sufficient to have AXA ■ A, and does not 
require that YZ is a full rank factorisation. Dually, (5.2) 
is both necessary and sufficient to have XAX * X, and BC 
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need not be a full rank'factorization. Observe, moreover, 
that given any matrix, A, with full ran* factorization A = BC, 
then for any choice* of *Y such' that' CY has full column rank, 
taking Z = (CY)^B^ with QCY) L any left inverse of CY and B 
any left inyerse of B gives a matrix X - YZ such that (5.2^ 
hdlds. Therefore we can always construct matrices,, X, of 
&ny given rank not exceeding the rank of A witfi XAX = X. On 
the other hand, given full rank factorizations A = BC and 
X - YZ such that*AXA » A and XAX ■ X, then for any matrix U 
with full column rank ^satisfying CU = 0 and for any matrix V 
with jfJVA defined we have — 

(5./) A(3frUV)A - A. 

Now 



(5.4) 



X + UV 



where the first matrix on the right-hand side has full column 
rank '(Exercise 5.6) . Thus, for any choice of V such that 
the second matrix on the right-hand side of (5.4) has full 



A). 



row rank, (5.3) holds and rank (X+UV) > rank (A) 

The following example illustrates the construction of 
matrices, X, of prescribed rank such that A and X satisfy at 
least one of the conditions AXA - X and XAX - X. 

Example 5.1 - 4 

Let A be the matrix 



0 
-3 
-1 



from Example 4.1 with full rank factorization 



BC 
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Then rank (A) - 2, and X Q 
To construct a matrix, X, 



-2' 
1 



Q satisfies X^AX^ 
oo 



of rank one such that XjAXj 



trivially. 
X, , 
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4 



(S.S) 



B L-7 



note first that 

■1 2 . 0 

2 -1 0_ 

is a left inverse of B. .Now if y H - {3 4 5], then 

21 J ' ■ JM-i n. z H -4 




, (Cy) 



so that 



x i - m 



SJ -33 0* 
76 -44 0 
95 -55. 0 



To next construct a raatrix s ^C % of rank two such that X 2 AX 2 » X 



(and thus AX 2 A « A) t let 
"1 -ll 

Y « 



Then 



CY 



1 1 
1 -1 



0 4 
5 -3 




and with the left inverse of B in (5.5), 

" 5 2 01 



-5 'lo 0 



so that 



X 2 ' YZ " JO 



"5-4 0 
0 6 0 
5-4 0 



Finally, to sonstruct a matrix , V X 3> of rank three sue/ that 



* A, let 




S 






" 2" 




"0" 


u ■ 


-3 


• v * OT 


0 




3_ 




1. 



9 
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where ueN(C). Then 



-4 2 
6 .-3 
-4 -3 



with, det X- - -5, 



That the procedure in 'Example 5 .1 can be extended to 
construct matrices -X of given rank satisfying (AX) H - and 
at least one of the conditions AXA - A and XAX * X is 
apparent by observing that CY with full CjDlumri^ralvk^impUes 
BCY has full column rank. Hence, taking Z «(BCY) + ," (5.2) 
holds and AX ■ BCY(BCY) + is' Hermitian; In^tlje following 
example we indicate matrices Z in .Example 5*1 so thar the 
resulting matrices X £ satisfy (AX i ) H - AX^ i - 1,2,3." * ■ 
Example 5.2 v * „ 

Given the matrix A anci full rank factorization A ■ Bd 
in Example 5.1, again let y H - [3 4 5]. Then 

• ' f34l 



Ay ■ BCy 



and 



"51 21 24 
69, 28 32 
8S- °3S ' 40 



satisfies with AX X Hermitian, and rank (Xj) * 1# 



Gfcntinuinj 



Y - 



if we again use 



then 



1 



AY - BCY 



\ i 

"10 -21 

S* 5 

5 1 



> Z - (AY) + 



1 



r 16 5 7 
-20 35 
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and 



YZ 



• rnr 



"18 


-15 






-2 


20 


6 




IS 


Z 1 * 


1 





with X 2 AX 2 » > X 2 ,AX 2 Hermitian and rank (X* 2 ) a \2. 




Now taking u 
H 



H 



[2 



-3] 



'nd 1/110 [0 1 »1] gives 



in the previous example 





"18 


-13 


3" 


■A 


-2 


17 


3 




18 


-18 


-2 



with AXjA - A, AX 3 Hermitian and det X 3 « -10. « x y 

* Given any full rank factorization A - BC, fjprst choosing 
a matrix Z so that ZB (and thus ZBC) has full row rank pro- 
/ vides a completely dual procedure to that in Example 3 5.1 in 
which Y * C R (ZB) R with C R any right inverse of C and (ZB) R 
an/ right inverse of ZB. faking Y - (ZBC)* then gives 
matrices analogous to* those in Example 5^2^in wlrich we? now. 
T have (X.A) H » X £ A, i « 1,2,3. 

* fJ We^conclude this brief introduction to generalized 
inverses that are not unique by observing that the question 
Ipf ^represenVing. all s**utions of particular subsets of equd- 

• •tionS such as AXA * A or* XAX ■ X and AX or XA Hermitian has 

not been considered. Also, although 6bvio.us properties of 
matrices A and X satisfying AXA ■ A witft*AX ancL XA Hermitian 
% 4 are included in the exercises,, the more difficult question 

* ''when AXA « A ft replaced by the nonlinear relation XAX • X, 

is only treated superficially. The interested reader is 
gorged to consult [2] for a detailed ^discussion of these 
topics . . J - z * 



* Exercises 

5.1 Show that any two, of the copdltlpns AXA - A, XAX - X, 
rank. (X) - rank (A) Imply the third. 



5.2. Show that XAX - A* If AXA - A, (AX) H - AX and (XA) n - XA 
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5.3 



Let A ■ BC and X YZ where t and 2 H have full column rank. 

H 



a. 



Show that XAX i X and (AX)" » AX i f and only if BCY ■ i 
Dually, show that^XAX - X and,(XA) H '- XA if and only if 



Z + . 



. ZBC - Y" . 
Given the matrix 



: AX, 



Xj and 



construct a matrix X, of rank one such tr, 
(AX,) - AX,. Also, construct a n*jta<xX 2 of rank two such 
that X AX 2 - X and (X 7 A) H 



5ik Cet A - 6C and«X YZ whefre , 
cfolumn rank. 



Is square and B and C H "have full 



Show that^T AXA - A and XA - AX, then B - YZBCB and 
C - JtttYZ where (CB)" 1 exists. 

'Why is It not of interest ta consider also the special ' 
cases when (ZB)' 1 or (QY)' 1 exist? 



c What equations must Y and Z satisfy if XAX « X and AX » 
5.5 Verify that the inverses constructed in Examples 5.1 and 5.2 

^satisfy the requi re<f properties . 

■» , 

[YJU^is any matrix wj th, CY nonsingular and 



XA? 



5*6 Pro^ that if W 

^ cqiumhs of U in N(C), linearly independent, then W has full column 



rank. 



• 5.7 ..Prove th* .If A - BC, is any' full rank factorization of a square 
matrix, -then CB - I If and only if.A is /fdonpotent. 

' 5.8' Show that If A - BC Is any full rank factorization and Y is any 
matrix sirch that CY,ls nonsingular! A + - [(AY) + a| + (AY) + .' 
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Appendix 1: 
Hints for Certain Exercises 



Chapter 1 



■ Ex * Klb: ,x y H " Y* H Implies y - ax where 



Ex. 1. 2c 



a.-feiXLuo. 

I ML 2 



axx" - axx H . f 
\f BA H A i 



CA R A then (BA H rCA H )A(B H -C H ) - 0. 



EX '.'- ?a ! ,f P Is thc ■»«"« w[ th columns x,,.'..,x. and A Is the, 
diagonal matrJx with diagonal e.ements A,, 1 - ,,..;,„, „ . PA . Hepce 
A - (PA)P sJnce P Is unitary. KJc: |f A Is Hermltian, X.E, - A,E * 
« - I,...,n. ' . II 

— '^ b ! . lf u k ls the co,um " vector with k elements' each.equal to 
unity, 



u ri-l 



for .11 n i 2. Kgc:* Subtract the last row of A„ from each of 



L 



the 
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preceding rows and expand the determinant using cof actors of the first 

* • -1 * 

column. 1 .5d : A n has all integral elements. 

' Ex. 1.6a : .Let X » I + 3xx H and determine 3 so that AX = I. \ .6b : 

where x - u..^. 1 .6e: Ax = (l+k)x and Ay = y. 



-611+ —xx I 



1 ,6f : For any n > 2 the vectors 



Y 

i 




' l" 




' r 




1 








-1 

0 




i 

-2 














J 


0 


7 • • • 7 






j 
















l 




0 




0 


.J 


4n-l). 





are orthogonal . 



Ex. 1.7b: Form XA first. 



Chapter 2 « 
Ex. 2.2 : A2 « 0, and-Zct =* o implies a » 0. 

Ext 2.5 : Xj is orthogonal to every vector z£N(A). Hence 

i nr 2 - 1 1*, ii*? i«,i 2 ♦ i« 2 i 2 - 



Ex. 2.6 : Let A be m by n with rank r, so that dim N(A) - n-r. Now assume 
rank (A H A) - k < r, and let z^...,:^ denote any basis of N(A A). Then 

0 « (ZjA.); (Az.,Az.) - llAz.il 2 - ■ $ 

uplies Azj - 0, i »yl/...,n-k. Hence dim N(A) >. n-k > nV , a 
ictlon. */ \ \ * 



impl 
d 



contra- 



Ex. 2.10 ; Use Exercise 2.9 and apply Exercise 1.6a. 

Ex. 2.11a : Use Exercise^. 10 toobta'in A + b and Exercise- 2.2 to fori* 

*eN(A). *2 ;11d : tn :thfs°case 

\ - » • 



Ex ' 2, '3 ; A " uv a full rank factorization. Use Theorem k and the 
remarks In the final paragraph of Section 2.2. 



x - A b + £a . 

1-1 1 



is an orfth< 



/ 



rfthogonai decomposition of any vector x. Now take *the inner/ 
product of x with any vector z.. ' , 

E *' 2 ; 15 : for any 1 " 1 !•••.«- column i of A + is the minimal norm solu- 
tion of Axj ■ e Jt 

EX ' 2 ' 20e : Usc ^rclse 1.2c and its dual that BAA H - CAA H if and only 
if BA - CA. 



Chapter 3 \ 

M 

* Ex. 3.2b : *A ^ 



[i W". l x] is a 



full rank factorization. 



F x - ?-7a: d k a k/ - 1 if and only if c . * 0. 



Ex, 3.11b : f / 

\ *+/ 1 

qlp+TT 



P P P . 

^ Now If x - A + y Is written in terms of 



V H 



co<y>onents as x* - '(i, tj . . ,t 



then 



and 



j ; "^iiji y u 



i - i,...,p. 



He: With ; <ilm M(A) « 1 antf z « 



-1 



>e written fn terms. of components as 



_fj,...,p t ^where a is arbitrary. 

• , 87 



£ N(A), ail solutions of Ax - y can 



- m - a and tj - tj +*a, 



If I t, 0, then 



f. I 

•I y-1 



m it j, y 'J + ^,U, y, j' 



i-i j-i 



so that 



and 



* .i is 



1 



1 



Ex. 3. 13 ; AxXpy - Aw(xXy). 

3. 14b: Let z.,...,z be any conjplete orthonormal set of eigenvectors 
* J n 

of I - WW where z, z„ correspond to eigenvalue A » 1 and ^ 

n J r , 

* r+ j ••••»*„ correspond to eigenvalue X » 0. Combine these vectors with 

those in the hint for Exercise 1 .6f . * 

— ° y x 

Ex. 3. 1 5 : Use Gauss el imi nation' to reduce T to blodfcform 



Ex. 3. 17 :* Zj XYj corresponds to eigenvalue one if and only if 



Ex. 3.20 : AX * p and XB D consistent implies AA C C and DB B • D. 



\ 
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Chapter 4 

Ex^jKZa: fcW - (C H B) + C H . k^t: The relation is reflexive, by 
# lemma «e) . If C is alias to B, then f - BB + C - B +H (B H C) is a full rank 
factorization and the relation is symmetric since (B +H ) + - B H . Transi- 
tivity follows by a similar type of argument. 

E *' *-3s With W » B H ,(B w ) d » B,(B H B) d 2 « B[(B H B ) + ] 2 . 

Ex^Jk_U: Use an argument similar to that one employed to establish 
uniqueness in (2.2). 

* • • 

Ex ' ^ i 8 lf A - BC is a full rank factorization and A has 'index one, 

A L ' C + \CB)' ] C and A R * B(CB)"U + , 
by Exercise Then 

A L A - C*C - A + A, Y - BfcBpB* and A n ^ - 0B + - AA + 
for all n > I . \ 

\ 

Ex^jKJi: Show first that X « B(WB) d P satisfies ail six equations. 
Then show that the first set of three>,equations implies the second set, 
and that the second set implies X has the >given form/ 

C *' *- t8 '^ f A d 2 - B d 2 then.A d B - AB^so that A^-'aX. and B<j - A^. 

* * * * // ' 

Chap ter 5 *« 

~ * v - 

Ex - 5-2: Use both the direct and dual form of Exercise 4.10 with I - 1 . 

Jx^lJa; Applying Exercise 4.10 to XAX - X and (AX) H - AX gives 
AX - X + X -Z + Z. 

Ex - 5.*a; B has fuli column' rank. 5.4b ; Then X • 
Ex - AY - B(CY) is a fuU rank factorization. 





y 
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